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ABSTRACT 


Current  assessment  doctrine  in  U.S.  Joint  Publications  needs  significant  reform  by 
drawing  on  civilian  practices  to  standardize  terminology  and  procedures  to  improve  its 
successful  use  in  future  operations.  Comparison  to  the  civilian  practice  of  assessment 
reveals  that  the  U.S.  military  incorrectly  defines  the  evaluation  process.  The  civilian 
practice  uses  evaluation  to  review  and  improve  the  operational  approach  and  elements  of 
operational  design,  update  the  assessment  plan,  and  validate  planning  assumptions.  Also, 
current  doctrine  does  not  provide  adequate  description  of  assessment  methodology  and 
lacks  common  procedures,  training,  subject  matter  experts  and  alignment  with  a  lead 
Joint  Staff  directorate.  The  lack  of  detail  about  assessment  sharing  between 
organizational  levels,  assessment  output  product  description,  and  traditional  warfare 
requirements  further  exacerbates  this  failure.  The  military  should  correctly  redefine  the 
monitoring  and  evaluating  processes  in  order  to  shift  the  focus  of  evaluation  from 
assessing  plan  execution  to  evaluating  the  plan  assumptions  and  objectives  for  continued 
synchronization  with  strategic  goals  and  changing  environment.  Moreover,  improvement 
of  the  U.S.  military  practice  of  assessment  requires  clearly  defining  doctrine, 
organization,  and  procedures  while  establishing  common  manning  and  training.  By 
implementing  these  changes,  the  U.S.  military  can  significantly  improve  its  ability  to 
assess  current  military  campaigns  and  operations. 
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CHAPTER  1:  INTRODUCTION 


Why  has  the  United  States  been  so  successful  in  recent  wars  and 
encountered  so  much  difficulty  in  securing  its  political  aims  after  the 
shooting  stops? 

-  Frederick  Kagan,  “War  and  Aftermath” 

When  he  took  command  of  ISAF,  General  Stanley  McChrystal  published  an 

initial  assessment  document  stating  the  following  about  assessment: 

ISAF  must  develop  effective  assessment  architectures... to  measure  the 
effects  of  strategy,  assess  progress  toward  key  objectives,  and  make 
necessary  adjustments.  ISAF  must  identify  and  refine  appropriate 
indicators  to  assess  progress,  clarifying  the  difference  between  operational 
measures  of  effectiveness  critical  for  practitioners  on  the  ground  and 
strategic  measures  more  appropriate  to  national  capitals.^ 

This  statement  implies  the  fact  that  ISAF  conducted  eight  years  of  war  without 

having  an  adequate  assessment  process  in  place  to  evaluate  the  campaign  against 

national  objectives  and  ensure  proper  alignment  with  national  goals.  The 

potentially  catastrophic  failure  of  the  assessment  system  to  ensure  the  continuous 

match  between  shifting  strategic  desires  and  operational  goals  is  due  to  a  gap  in 

the  current  US  assessment  methodology.  Joint  Publication  3-0  Joint  Operations 

presciently  states,  “changes  to  the  military  objectives  may  occur  because  political 

and  military  leaders  gain  a  better  understanding  of  the  situation,  or  they  may 

occur  because  the  situation  itself  changes.  The  Joint  Force  Commander  (JFC) 

should  anticipate  these  shifts  in  political  goals  necessitating  changes  in  military 


*  Stanley  McChrystal,  “COMISAF  Initial  Assessment  (Unclassified),”  The  Washington  Post, 
September  21,  2009,  http://www.washingtonpost.com/wp- 

dvn/content/article/2009/09/21/AR20090921001  lO.html  (accessed  February  27,  2014). 

^  Jonathan  Schroden,  "Why  Operations  Assessment  Fails:  It's  Not  just  the  Metrics,"  Naval  War 
College  Review  64,  no.  4  (Autumn  2011),  90. 
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objectives.”  While  the  JFC  looks  for  these  changes  during  interactions  with  the 
President  and  Secretary  of  Defense,  the  JFC  also  has  a  staff  that  is  responsible  for 
monitoring  the  progress  of  the  war  and  looking  for  changes  as  well. 

Unfortunately,  current  U.S.  assessment  methodology  does  not  include  a  system 
for  monitoring  the  relevance  of  the  elements  of  operational  design  and  evaluating 
their  alignment  with  political  goals. 

The  thesis  of  this  paper  is  that  current  assessment  doctrine  in  U.S.  Joint 
Publications  needs  significant  reform  and  should  draw  on  civilian  theory  and  practices  to 
standardize  terminology  and  procedures  to  improve  its  successful  use  in  future  joint 
operations.  JFCs  are  responsible  for  waging  military  campaigns  in  order  to  achieve 
operational  objectives  that  support  strategic  objectives.  JFCs  need  to  assess  the 
progression  of  their  campaign  so  they  can  appropriately  adjust  to  changes  in  the 
operational  environment  and  respond  to  enemy  actions.  Assessment  is  the  current 
doctrinal  methodology  that  attempts  to  provide  that  vital  feedback  to  the  JFC.  However, 
current  assessment  methodology  is  not  working. 

Unfortunately,  current  assessment  doctrine  is  generally  convoluted  and 
inadequate  to  such  an  extent  that  the  assessments  provided  to  JFCs  can  be  erroneous, 
misleading,  or  inconclusive.  The  bigger  failure  is  that  assessment  doctrine  does  not 
instruct  commanders  and  assessment  teams  on  how  to  evaluate  potential  shifts  in  the 
elements  of  operational  design  such  as  strategic  end-states  or  objectives.  Instead,  it 
focuses  almost  all  of  its  attention  on  monitoring  the  execution  of  the  current  plan. 


^  U.S.  Joint  Chiefs  of  Staff,  Joint  Operations,  Joint  Publication  3-0  (Washington  DC:  Joint  Chiefs 
of  Staff,  August  11,  2011),  A- 1. 
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informing  the  commander  “If  we  are  doing  the  right  things?”  While  this  is  an  important 
measure  of  campaign  progress,  beyond  paying  lip-service  to  the  fact  that  the  JFC  should 
be  aware  that  the  operational  approach  may  require  revision  during  an  operation,  there  is 
no  methodology  in  joint  doctrine  that  assessment  teams  can  follow  to  evaluate  if  the 
“right  things”  that  were  initially  identified  are  still  valid. 

In  order  to  understand  why  current  assessment  is  failing  and  how  to  improve  it,  a 
review  of  the  civilian  practice  of  assessment  will  show  that  the  military  fails  to  define 
correctly  and  properly  separate  the  assessment  concepts  of  monitoring  and  evaluation. 
Proper  separation  creates  an  evaluation  element  that  is  responsible  for  focusing  on  plan 
refinement,  updating  the  assessment  plan  itself,  and  assessing  the  assumptions  associated 
with  the  plan.  Subsequently,  a  review  of  current  assessment  definitions,  doctrine,  and 
practices  will  show  that  current  doctrine  does  not  provide  adequate  definition  of 
assessment  terms,  practices,  or  methodology  including  a  lack  of  emphasis  on  subject 
matter  experts,  standardized  products,  and  unique  traditional  warfare  considerations.^ 
Finally,  recommendations  for  changes  to  assessment  doctrine  include  creating  distinct 
monitoring  and  evaluation  assessment  teams  aligned  with  separate  directorates,  as  well  as 
producing  more  robust  practices  and  procedures.  Making  these  changes  to  assessment 
methodology  will  establish  an  assessment  system  that  is  able  to  view  the  campaign 
holistically  up  to  the  strategic  level  and  provide  the  JFC  with  recommendations  on 
campaign  changes.  This  shift  will  allow  the  U.S.  military  to  achieve  the  highly  elusive 

U.S.  Joint  Chiefs  of  Staff,  Joint  Operation  Planning,  Joint  Publication  5-0  (Washington  DC; 
Joint  Chiefs  of  Staff,  August  1 1,  201 1),  D-3. 

^  Current  joint  doctrine  definitions  separate  warfare  into  two  types;  traditional  and  irregular. 
Traditional  warfare  comprises  large-scale  conventional  campaigns  and  operations.  Irregular  warfare  is  all 
other  operations  such  as  counter-insurgency  and  counter-terrorism  operations.  For  further  information,  see 
JP3-0,  page  1-14. 
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long-term  success  in  achieving  strategic  objectives  and  end-states,  despite  the  fact  that 
they  may  shift  throughout  the  campaign. 

An  extensive  literature  study  indicates  that  current  assessment  procedures 
generally  fail  to  meet  the  requirements  of  the  JFC.^  From  this  start  point,  it  is  possible  to 
focus  instead  on  the  root  flaws  in  the  current  system  and  potential  remedies  available 
without  needing  to  establish  the  inadequacy  of  the  current  system.  Also,  both  the  U.S. 
and  NATO  are  currently  producing  a  large  amount  of  literature  and  research  in  the  area 
of  assessment.  The  two  agencies  are  working  on  similar  and  complementary 
improvement  programs  that  inform  each  other  as  they  progress.  This  paper  will  evaluate 
the  current  state  of  U.S.  doctrine,  but  it  will  consider  both  U.S.  and  NATO  written 
procedures  when  considering  the  inadequacy  of  current  procedures  and  the  establishment 
of  potential  remedies. 


®  Numerous  assessment  experts  have  written  extensively  in  professional  literature  about  the  failed 
state  of  current  assessment  practices  as  used  in  Iraq  and  Afghanistan.  At  least  five  of  the  articles  in  the 
bibliography  describe  the  failure  of  assessment.  Reference  the  publications  by  Cordesman,  Downes- 
Martin,  MacIntyre,  Schroden  and  Sotire  for  further  details. 
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CHAPTER  2:  CURRENT  ASSESSMENT  DOCTRINE 


What  gets  measured  gets  managed,  and  what  gets  managed  gets  done. 

-  Dean  Spitzer,  Transforming  Performance  Measurement 

Assessment  Defined 

Joint  Publication  3-0  explains  that  Joint  Force  Commanders  conduct  joint 
operations  by  accomplishing  four  separate  procedures:  planning,  preparing,  executing 
and  assessing.*  JP  5-0  describes  assessment  as  “the  continuous  monitoring  and 
evaluation  of  the  current  situation  and  progress  of  a  joint  operation  toward  mission 
accomplishment.”  The  purpose  of  assessment  is  to  inform  commanders  if  their  plan  is 
working  or  if  it  needs  to  be  refined.  In  order  to  achieve  this  requirement,  assessments 
generally  attempt  to  compare  the  current  progress  of  the  campaign  to  the  advancement 
predictions  made  during  planning.  According  to  NATO  procedures,  the  assessment 
process  has  four  steps:  assessment  plan  design,  data  collection  plan  development,  data 
collection  and  treatment,  and  analysis,  interpretation,  and  recommendations.  By 
following  these  four  steps,  an  assessment  team  (AT)  is  able  to  provide  a  commander  with 
an  assessment  of  campaign  progress.  Assessments  then  should  provide  the  bridge 
between  the  military  strategy  that  is  the  focus  of  the  campaign  and  the  truth  of  the  current 
context  of  the  war  by  fostering  recommendations  for  strategy  and  campaign  plan 
improvement.'^ 


*  U.S.  Joint  Chiefs  of  Staff,  Joint  Operations,  Joint  Publication  3-0  (Washington  DC;  Joint  Chiefs 
of  Staff,  August  11,2011),  I-l. 

^  U.S.  Joint  Chiefs  of  Staff,  Joint  Operation  Planning,  Joint  Publication  5-0  (Washington  DC: 
Joint  Chiefs  of  Staff,  August  1 1,  201 1),  III-44. 

^  NATO  Operations  Assessment  Sub-Working  Group,  NATO  Operations  Assessment  Handbook, 
(Norfolk,  VA:  NATO  Supreme  Allied  Commander  Transformation,  2012),  2-4. 

Ethan  B.  Kapstein,  “Military  Metrics:  How  Do  We  Know  When  We’re  Winning  (Or  Losing)  a 
War?”  Smallwarsjournal.com,  July  6,  2011,  http://smallwarsiournal.com/blog/iournal/docs-temp/806- 
kapstein.pdf  (accessed  August  12,  2013),  2. 
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A  Short  History  of  Assessment  Development 


The  creation  and  development  of  assessment  in  the  military  closely  related  to  the 
rise  of  the  concepts  of  maneuverist  approach  and  effects-based  operations  (EBO).  In 
World  War  II,  the  development  of  aircraft  and  other  distance  technologies  allowed 
commanders  to  attempt  to  defeat  the  enemy  without  directly  destroying  the  enemy 
military.^  Various  operations,  such  as  the  Combined  Bomber  Offensive  attack  on  the 
ball-bearing  plants  in  Germany,  attempted  to  achieve  this.  The  problem  with  this 
approach  is  that  it  was  difficult  to  measure  the  produced  effects  because  they  are  not  as 
obvious  as  lines  on  a  map  moving  to  reflect  a  front  line  gain  or  ground  lost.  In  the  1950s, 
the  renowned  management  consultant  Peter  Drucker  produced  his  seminal  work  on  the 
philosophy  of  Resource-Based  Management  (RBM)  which  was  gradually,  albeit 
haphazardly,  implemented  by  the  military  in  various  forms. ^  RBM’s  core  concept  is  to 
shift  the  focus  of  planning,  managing,  and  decision-making  from  inputs  and  processes  to 

n 

the  desired  outcomes  and  results.  This  approach  requires  a  company  to  lay  out  objectives 
and  postulate  logical  linkages  to  connect  the  required  means  with  the  desired  ends.  Thus, 
an  important  part  of  this  theory  is  the  ability  to  measure  the  outcomes  generated  to  ensure 
that  the  plan  is  succeeding.  Since  1991,  assessment  gained  importance  in  the  military 
campaign  system. 

Current  Assessment  Doctrine 

IP  3-0  and  IP  5-0  are  the  current  U.S.  military  doctrine  publications  that  define 
the  methodology  for  assessment.  The  assessment  description  contained  in  IP  3-0  is 

^  David  S.  Dees,  Effects-Based  Operations  at  the  Operational  Level  of  War:  Exploring  the  Living 
System  Alternative  (Newport,  RI;  Naval  War  College,  2008),  2. 

®  NATO  Operations  Assessment  Sub-Working  Group,  6-68. 

’  Ibid.  6-69. 
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mostly  duplication  of  the  JP  5-0  assessment  information.  JP  5-0  instructs,  “Assessments 
should  answer  two  questions:  Is  the  JFC  doing  things  right?  Is  the  JFC  doing  the  right 

o 

things?”  To  properly  answer  these  questions,  assessment  teams  (ATs)  create,  collect  and 
analyze  Measures  of  Performance  (MOP)  and  Measures  of  Effectiveness  (MOE).^  MOPs 
are  used  to  assess  friendly  accomplishment  of  assigned  tasks  and  help  answer  the 
question  “are  we  doing  things  right?”'°  MOEs  enable  the  assessment  of  changes  in 
system  behavior,  capability,  or  environment  toward  achieving  end  states  or  objectives 
and  answer  the  question  “are  we  doing  the  right  things?”^ ^  One  of  the  real  values  in  this 
assessment  methodology  comes  from  comparing  the  MOPs  and  MOEs  to  see  if  the 
actions  that  the  Joint  Eorce  is  executing  are  leading  to  the  desired  effects.  By  comparing 
these  two  metrics  across  the  range  of  desired  effects  and  tasks,  the  commander  will  know 
if  the  campaign  is  proceeding  as  predicted  and  desired,  or  if  alterations  are  required  to 
improve  mission  accomplishment. 

JP  5-0  continues  with  a  description  of  the  assessment  process  in  annex  D  which  is 
dedicated  solely  to  the  assessment  methodology.  This  annex  explains  that  the  purpose 
of  assessment  is  deliberately  to  compare  forecasted  outcomes  to  actual  results  in  order  to 
determine  the  effectiveness  of  the  military  operation  or  campaign.  The  assessment 
process  is  then  explained  in  three  parts  consisting  of  monitoring  the  current  situation  by 
collecting  relevant  information,  evaluating  the  data  by  using  criteria  to  judge  progress 


*  U.S.  Joint  Chiefs  of  Staff,  Joint  Operation  Planning,  III-44. 
^  Ibid. 

Ibid.,  III-45. 

"  Ibid. 

'^Ibid.,  D-1. 
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toward  established  goals  and  then  recommending  actions  or  changes  to  the  From 

there,  the  annex  devolves  into  a  discussion  of  combat  assessment  composed  of  battle 
damage  assessment,  munitions  effectiveness  assessment  and  targeting  considerations,  all 
of  which  focus  on  the  tactical  and  MOP  level/^  Combat  assessment  has  very  specific 
uses  in  traditional,  full-scale,  kinetic  military  operations,  but  it  has  less  applicability  in 
irregular  warfare  or  non-kinetic  effects.  The  total  discussion  of  assessment  in  JP  5-0,  the 
primary  doctrine  on  assessment  is  a  mere  13  pages.  There  is  also  assessment  information 
in  seven  other  JP  3-series  documents  that  provides  limited  information  on  assessing 
specific  types  of  operations. Still,  this  is  a  paltry  amount  of  information  on  assessment 
when  compared  to  the  hundreds  of  pages  in  joint  publications  dedicated  to  the  other  joint 
operations  procedures  of  planning,  preparation  and  execution.  The  joint  community 
needs  to  provide  more  doctrinal  guidance  for  assessment  in  order  to  improve  its 
evaluation  of  military  operations. 


Ibid.,  D-5. 

Ibid.,  D-7. 

U.S.  Joint  Staff,  J-7,  Commander’s  Handbook  for  Assessment  Planning  and  Execution,  (Suffolk, 
VA:  Joint  Staff,  J-7,  Joint  and  Coalition  Warfighting,  September  9,  201 1),  VI-2. 


CHAPTER  3:  FAILURE  TO  SEPARATE  MONITORING  AND  EVALUATION 


A  common  mistake  that  many  leaders  make  is  to  allow  themselves  to 
become  too  engrossed  in  the  details,  too  fascinated  by  the  tactical  aspects 
of  the  enterprise... The  toughest  job  for  the  leader,  then,  is  to  trust  in  the 
strategy,  trust  in  the  subordinate  leaders,  and  trust  in  the  sensors  to  do 
their  jobs  to  report  the  right  information;  in  doing  so,  they  should  be  able 
to  stay  out  of  the  thicket  of  tactical  execution. 

-  ADM  James  G.  Stavridis,  Partnership  for  the  Americas 

Current  Doctrine  Incorrectly  Defines  Monitoring  and  Evaluation 

Current  doctrine  does  not  correctly  describe  the  assessment  concepts  of 
monitoring  and  evaluation  in  accordance  with  the  theoretical  concepts  of  Resource-Based 
Management  (RBM)  or  Theory-Driven  Evaluation  that  were  the  basis  for  the  current 
assessment  practices  and  doctrine.  This  improper  definition  is  a  primary  cause  of  the 
failure  of  current  assessment  practices.  The  basic  concept  of  RBM  is  that  the  most 
important  focus  of  a  business  is  not  on  maximizing  the  quality  or  quantity  of  the  product 
produced,  but  rather  lies  in  the  selection  of  the  correct  and  best  objectives  for  achieving 
their  aims  and  then  assessing  them  to  see  if  their  attainment  is  leading  to  the  ultimate 
goal.'  For  example,  most  companies  want  to  make  money  and  there  are  many  ways  they 
can  set  about  accomplishing  this  goal.  They  can  build  more  products,  build  better 
products,  build  different  models  of  the  product,  or  expand  the  number  of  factories 
producing  the  product.  Each  of  these  objectives  could  hypothetically  increase  the 
revenue  of  the  company  by  selecting  one  or  an  amalgamation  of  several  as  its  strategy  to 
attain  its  goal.  Once  selected,  the  company  management  then  assesses  their  operations  by 
monitoring  the  achievement  of  the  objective;  are  they  producing  more,  better,  or  diverse 
products.  But  more  importantly,  evaluating  of  the  objective  also  ensures  that  it  is 

*  NATO  Operations  Assessment  Sub-Working  Group,  NATO  Operations  Assessment  Handbook, 
(Norfolk,  VA:  NATO  Supreme  Allied  Commander  Transformation,  2012),  2-7.,  6-68. 
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producing  the  desired  increase  in  revenue.  From  RBM  theory,  the  military  needs  to  learn 
that  without  evaluating  the  selected  operational  design  elements,  such  as  end-state  or 
objective,  to  ensure  they  will  in  fact  produce  the  desired  strategic  outcome,  military 
operations  will  have  a  diminished  chance  of  being  successful  in  the  strategic  context. 

JP  5-0  identifies  assessment  as  a  three  step  process  consisting  of  monitoring, 
evaluation,  and  recommendation.  The  definition  of  monitoring  is  the  collection  of  data 
and  metrics  to  analyze  the  current  situation.  Evaluation  is  determining  progress  toward 
end-states,  objectives  and  tasks.  Recommendation  is  the  use  of  monitoring  and 
evaluation  to  adjust  the  plan  execution.  Unfortunately,  definitions  of  monitoring, 
evaluation,  and  recommendation  focus  all  the  assessment  effort  on  evaluating  the 
execution  of  the  approved  plan. 

The  root  cause  of  this  misdirected  focus  on  plan  execution  is  assessment’s 
reliance  on  Measures  of  Performance  (MOPs)  and  Measures  of  Effect  (MOEs)  as  the 
only  tools  provided  for  the  assessment  process.  Both  tools  focus  on  measuring  the  proper 
execution  of  the  plan  and  determining  if  the  plan  is  accomplishing  the  selected  goals.  JP 
5-0  states  that  MOPs  measure  “is  the  JFC  doing  things  right?”  and  MOEs  measure  “is  the 
JFC  doing  the  right  things?”'^  Both  of  these  questions  assume  that  the  JEC  has  correctly 
determined  key  elements  of  operational  design  such  as  end-state,  objectives,  and  centers 
of  gravity.  Eurther,  the  current  methods  of  measuring  the  plan  do  not  track  or  identify 
any  changes  in  those  key  factors  or  the  context  in  which  the  staff  identified  them.  This  is 
due  to  the  fact  that  MOPs  and  MOEs  are  not  appropriate  for  monitoring  the  relevance  of 

^  U.S.  Joint  Chiefs  of  Staff,  Joint  Operation  Planning,  Joint  Publication  5-0  (Washington  DC; 

Joint  Chiefs  of  Staff,  August  1 1,  201 1),  D-2. 

^  Ibid. 

Ibid.,  III-44. 
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selected  elements  of  operational  design  like  centers  of  gravity  or  end- states.  By  relying 
on  MOPs  and  MOEs,  the  current  doctrinal  definitions  of  monitoring  and  evaluation  focus 
assessment  exclusively  on  determining  if  the  plan  is  achieving  the  originally  determined 
operational  approach. 

JP  5-0^  and  the  U.S.  Assessment  Handbook^  caution  the  military  practioner  to  be 
aware  that  the  initial  conditions  and  assumptions  used  to  build  a  campaign  are  subject  to 
change.  Specifically,  JP  5-0  details  that  it  is  the  responsibility  of  the  J-5  future  planners 
to  evaluate  if  the  assumptions  and  desired  end-states  remain  correct  and  desirable.  If  it  is 
determined  they  are  lacking,  the  planners  should  reframe  the  problem  and  consider  a 

•7 

branch  plan  or  new  Course  of  Action.  As  a  campaign  progresses,  it  is  not  just  likely,  but 
almost  assured  that  everything  about  the  operating  environment  and  possibly  even  the 
strategic  guidance  will  change  including  the  specifics  of  the  objectives  or  the  center  of 
gravity.  The  strategic  end-state  is  likely  to  shift  as  the  plan  also  meets  political  reality 
that  will  cause  a  change  in  both  the  military  end-state  and  desired  termination  criteria. 
Unfortunately,  while  doctrine  identifies  the  need  to  monitor  assumptions  and  end-states, 
and  for  the  future  planners  to  reframe  the  problem,  it  does  not  provide  “how-to”  guidance 
for  monitoring  these  areas  to  identify  changes  and  make  the  appropriate  adjustments  in 
the  plan.  The  fact  that  doctrine  mentions  the  reframing  concept  separately  from  the 
assessment  process  and  has  little  discussion  on  what  would  constitute  reframing  trigger 
events  highlights  this  lack  of  guidance. 

The  beginning  of  potential  remedies  to  the  assessment  problem  lies  in  the  NATO 

^  Ibid.,  III-23. 

®  U.S.  Joint  Staff,  J-7,  Commander’s  Handbook  for  Assessment  Planning  and  Execution,  (Suffolk, 
VA;  Joint  Staff,  J-7,  Joint  and  Coalition  Warfighting,  September  9,  201 1),  1-6. 

’ibid.,  11-18. 
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Operations  Assessment  Handbook.  This  handbook  identifies  that  assessment  is 
responsible  for  two  independent  processes:  operational  assessment  focused  on  the  short 
to  mid-term  accomplishment  of  effects  and  decisive  points,  and  campaign  assessment 
focused  on  evaluating  if  the  objectives  and  effects  are  accomplishing  the  military 

o 

mission.  Current  doctrine  lacks  procedures  and  a  specific  group  responsible  for 
selecting  the  reframing  indicators,  monitoring  the  strategic  environment,  and  evaluating 
the  operational  approach  to  determine  if  it  is  still  valid  or  requires  reframing. 

The  Need  to  Separate  Evaluation  from  Monitoring 

Clausewitz  famously  states  that  “war  is  merely  the  continuation  of  policy  by  other 
means’’^  denoting  all  wars  are  inextricably  linked  to  the  achievement  of  political  aims. 
The  impact  of  this  reality  is  that  it  is  very  possible  to  meet  all  original  military  objectives 
without  achieving  the  political  goals  of  the  nation  as  they  evolve  over  time.  Most  people 
immediately  consider  the  U.S.  operations  in  Vietnam  or  Iraq  as  examples  of  not  meeting 
the  political  objectives,  but  it  has  actually  plagued  militaries  throughout  history.  Pierre 
Lessard  points  out  that  probably  the  most  famous  example  of  this  is  when  Napoleon 
defeated  the  Russian  Army  and  captured  the  national  capital  in  1812,  but  was  unable  to 
fully  defeat  his  enemy  and  prevent  them  from  successfully  rebuilding  and  ultimately 
defeating  the  French  Army.'°  These  examples  highlight  the  fact  that  the  most  important 
part  of  strategy  and  military  operations  is  periodically  assessing  objectives  and  end-states 
to  ensure  alignment  with  policy  aims  in  the  changing  context  of  the  strategic 

*  NATO  Operations  Assessment  Sub-Working  Group,  2-7. 

^  Carl  Von  Clausewitz,  On  War.  ed.  and  trans.  Michael  Howard  and  Peter  Paret.  (Princeton,  NJ; 
Princeton  University  Press,  1976),  87. 

***  Pierre  Lessard,  “Campaign  Design  for  Winning  the  War... and  the  Peace,”  Parameters  35,  no.  2 
(Summer  2005),  36. 
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environment.  Unfortunately,  current  assessment  practices  in  doctrine  assume  that  the 
original  plan  or  policy  aims  are  correct  and  the  goal  of  assessment  is  to  help  ensure  the 
proper  execution  of  the  plan.  This  leads  to  the  major  logic  shortfall  that  assumes  the 
original  understanding  of  the  operating  environment  matches  current  reality  and  the 
purpose  of  assessment  is  to  recommend  changes  to  actions  to  create  the  selected  effects. 
Assessment  needs  to  be  adapted  to  evaluate  the  key  aspects  of  the  operational  approach 
and  design  to  provide  relevant  reframing  indicators  and  recommendations  to  the 
commander. 

In  order  to  gain  a  better  understanding  of  how  to  improve  assessment  practices, 
the  U.S.  military  can  turn  to  civilian  assessment  practices  and  theories.  Interestingly, 
Theory-Driven  Evaluation,  which  is  the  basic  concept  underlying  modern  business 
assessment  practices,  provides  a  mechanism  to  do  just  that.  As  previously  discussed,  the 
private  sector  and  government  organizations  developed  and  successfully  employed 
assessment  methods  for  decades  and  culminated  in  the  concept  of  Theory-Driven 
Evaluation.  This  theory  contains  many  of  the  same  concepts  and  definitions  that  are 
enshrined  in  current  military  assessment  doctrine  and  the  U.S.  assessment  handbook. 
Unfortunately,  the  military  adopted  this  method  without  detailed  theoretical  or 
philosophical  understanding,  which  led  to  a  glaring  misapplication  of  the  monitoring 
and  evaluation  concepts. 

The  root  problem  lies  in  the  fact  that  the  military  misidentified  the  civilian 

**  Clinton  R.  Clark  and  Timothy  J.  Cook,  "A  Practical  Approach  to  Effects-Based  Operational 
Assessment,"  Air  &  Space  Power  Journal  22,  no.  2  (Summer  2008),  85. 

Andrew  P.  Williams  and  John  C.  Morris,  “The  Development  of  Theory -Driven  Evaluation  in 
the  Military:  Theory  on  the  Front  Line,”  American  Journal  of  Evaluation  30,  no.  1  (March  2009),  64. 

Ibid.,  63. 
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definitions  of  monitoring  and  evaluation.  The  military  used  both  monitoring  and 
evaluation  to  define  the  same  thing,  the  monitoring  of  mission  execution.  This  is  in 
contravention  to  the  philosophical  basis  of  theory-driven  evaluation  that  conceptualized 
two  main  domains  for  the  performance  of  evaluation:  normative  and  causative.  The 
normative  domain  seeks  to  determine  the  congruency  between  the  planned  program  and 
the  actual  outcomes  achieved.  This  is  the  plan-focused  assessment  that  military  doctrine 
currently  describes.  Alternatively,  there  is  the  causative  domain,  which  is  the  more 
important  component  in  theory-driven  evaluations.  The  causative  domain  expands  the 
evaluation  to  test  the  plan  itself  and  ensures  its  validity.  It  is  only  through  causative 
evaluation  that  it  is  possible  to  assess  a  plan  as  appropriate  to  the  conditions  that  currently 
exist  or  are  expected  to  occur. 

International  and  government  agencies,  other  than  the  U.S.  military,  have 
correctly  adopted  theory-driven  evaluation  and  are  generally  seeing  successful  use  of  its 
methods.  For  example,  the  United  Nations  Education,  Scientific,  and  Cultural 
Organization  (UNESCO)  has  been  using  the  theory,  under  the  title  Resource-Based 
Management  (RBM)  since  2005.  The  UNESCO  RBM  Guiding  Principles  pamphlet 
describes  how  it  conducts  the  concepts  of  monitoring  and  evaluation  separately. 
Monitoring  is  concentrated  on  assessing  progress  towards  the  desired  objectives.  In  other 
words,  it  focuses  on  ensuring  the  successful  execution  of  the  plan.  Evaluation,  in 
contrast,  moves  beyond  the  assessment  of  progress  and  seeks  to  determine  if  it  is  possible 
to  improve  the  plan  in  terms  of  relevance,  efficiency,  effectiveness,  impact,  and/or 

Ibid.,  64. 

Ibid.,  65. 

UNESCO  Bureau  of  Strategic  Planning,  Results-Based  Programming,  Management  and 
Monitoring  (RBM)  Approach  as  Applied  at  UNESCO  (Paris:  UNESCO,  201 1),  16. 
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sustainability.  The  UNESCO  pamphlet  maintains  the  most  important  concept  is  that 
different  teams  conduct  monitoring  and  evaluation  at  separate  levels.  The  project 
manager  leads  monitoring  and  is  directly  responsible  for  executing  the  plan.  Evaluation 
is  completely  independent  of  the  project  manager  and  conducted  at  the  top-level  of  the 
organization  that  oversees  all  UNESCO  programs.  By  solely  focusing  on  evaluating  the 
plan,  current  military  assessment  doctrine  misses  this  separation  of  monitoring  by  the 
plan  executors  from  a  different  group  that  evaluates  the  overall  plan. 

The  reason  for  the  military  misapplication  of  the  theory  is  readily  apparent  to 
anyone  who  visits  military  assessment  teams  (ATs).  The  teams  are  mostly  staffed  by 
military  officers  and  civilian  analysts  drawn  from  other  areas  of  the  organization.  This 
means  that  almost  none  of  them  have  formal  assessment  training  or  understand  equivalent 
civilian  practices  for  potential  incorporation  into  the  military  methodology.  Williams  and 
Morris  note  that  in  their  observations  of  various  ATs  in  the  military  they  have  “not  yet 
encountered  any  that  have  awareness  of  the  ‘civilian’  evaluation  field  -  a  fact  that  is  quite 
surprising  given  the  level  and  quantity  of  development  of  evaluation  technique  and  theory 
in  the  civilian  domain.”^^  The  military  can  learn  a  great  deal  from  the  civilian  sector 
about  properly  applying  theory-based  evaluation.  By  tapping  into  this  resource,  it  can 
radically  improve  the  accuracy  and  reliability  of  its  assessment  methodology. 

Reviewing  and  Improving  the  Selected  Plan 

Military  operations  take  place  in  complex  and  challenging  environments,  full  of 
tentative  alliances  where  each  participant  has  its  own  objectives.  They  also  consist  of  a 


Ibid.,  37. 

Williams  and  Morris,  68. 
Ibid. 
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variety  of  overlapping  and  sometimes  competing  missions  accomplished  simultaneously. 
The  military  often  views  and  analyzes  this  dynamic  operational  using  the  six  areas  of 
political,  economic,  social,  military,  infrastructure  and  information,  or  PMESII.  The 
complex  nature  of  the  domain  in  which  the  military  operation  is  taking  place  means  that 
it  is  difficult  to  ascertain  that  any  specific  tasks  will  lead  to  specific  effects.  In  fact,  in 
many  operations,  the  military  will  successfully  complete  all  assigned  tasks  and  still  have 
not  created  the  desired  effects  or  achieve  the  most  current  end-states.  Even  worse,  these 
complex  forces  will  often  work  to  shift  the  objectives  and  end-states  desired  by  both  sides 
as  the  operation  progresses,  especially  in  long-lasting  campaigns. 

Current  doctrine  is  aware  of  the  changing  nature  of  the  environment  during  plan 
execution.  In  several  places,  IP  5-0  notes  that  the  political  environment  will  likely 
change  during  the  execution,  which  often  results  in  a  change  to  the  military  mission  or 
the  desired  termination  criteria.  It  goes  on  to  say  that  when  these  changes  are 
significant  or  assumptions  proven  false,  the  commander  and  staff  need  to  reframe  the 
operational  approach.  Unfortunately,  doctrine  ends  with  those  statements  and  provides 
no  real  guide  or  instruction  on  what  the  commander  and  staff  should  do  to  monitor  and 
adapt  to  these  potential  changes. 

History  shows  that  selecting  and  achieving  the  strategic  level  goals  is  the  most 
important  part  of  military  action.  Williamson  Murray,  a  noted  military  historian  and 
strategic  commentator,  asserts,  “The  military  lessons  of  wars  in  the  twentieth  century 
suggest  that  it  is  at  the  political  and  strategic  levels  that  wars  are  won  or  lost  -  rarely  at 

NATO  Operations  Assessment  Sub-Working  Group,  2-1. 

Ibid.,  2-10. 

U.S.  Joint  Chiefs  of  Staff,  Joint  Operation  Planning,  D-5. 

^Ubid.,  11-18 -11-19 
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the  operational  and  never  at  the  taetieal.”  If  this  is  the  case,  then  the  most  important 
task  of  the  military  operational  commander  is  to  stay  in  touch  with  the  strategic 
leadership  and  ensure  that  the  military  executes  the  campaign  in  accord  with  the  strategic 
goals.  Bob  Woodward  points  out  that  the  last  60  years  of  military  history  is  replete  with 
examples  of  constantly  shifting  political  goals,  and  thus  “military  strategic  objectives  are 
rarely  enduring,  and  campaign  design  must  be  sufficiently  agile  to  adjust  to  their 
fluctuation.”  Even  World  War  II,  which  most  people  think  entailed  fairly  consistent 
policy  and  focus  from  the  Allies  perspective,  experienced  no  fewer  than  eight  major 
strategic  shifts  between  1942  and  1945,  averaging  a  change  every  5  months. The 
commander  should  expect  that  the  operational  approach  selected  at  the  beginning  of  a 
campaign  will  need  altering  as  the  military  operation  progresses. 

The  reality  of  politics  is  that  there  is  an  inextricable  linkage  between  them  and  the 
results  of  military  conflict.  As  national  leaders  face  consecutive  successes  or  failures  of  a 
campaign,  they  adapt  their  own  expectations  to  the  outcomes  the  nation  can  realistically 
achieve.  The  perfect  example  of  this  is  the  Korean  conflict  of  1950-1953.  During  a  war 
that  had  three  major  phases,  the  strategic  goals  of  the  allies  shifted  dramatically  over  the 
3  year  period.  First,  it  was  simply  to  maintain  control  of  any  part  of  South  Korea. 

Second,  after  significant  military  success,  the  political  goal  changed  to  defeating  North 
Korea  in  detail  and  uniting  the  country  under  democratic  control.  Finally,  after  China 
entered  the  conflict  and  rebuffed  the  Allied  advances  into  North  Korea,  it  transitioned 


Williamson  Murray  and  Kevin  Woods,  Thoughts  on  Effects-Based  Operations,  Strategy  and  the 
Conduct  of  War,  (Alexandria,  VA:  Institute  of  Defense  Analyses,  Joint  Advanced  Warfighting  Program, 
2004),  5. 

Lessard,  42. 
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again  to  re-establishing  two  separate  countries  in  a  status  quo  antebellum.  National 
leaders  must  always  balance  the  natural  security  interests  of  the  country  with  their  own 
political  goals  and  naturally  adapt  their  desired  objectives  based  on  the  current  success  or 
failure  of  a  military  campaign. 

The  assessment  team  must  be  responsible  for  creating  a  separate,  operational 
approach-focused  evaluation  to  provide  an  independent  and  holistic  assessment  of  the 
operation.  It  would  also  require  doctrine  to  define  the  process  and  methodology  for  how 
to  accomplish  this,  but  the  current  military  sanctioned  tools  of  MOPs  and  MOEs  cannot 
accomplish  it.  This  naturally  leads  back  to  an  analysis  of  the  civilian  theory  and  practice 
of  evaluation  to  see  if  there  are  tools  and  concepts  the  military  can  adapt  for  its  use. 

The  genesis  of  assessment  theory  comes  from  Theory-Driven  Evaluation.  The 
central  tenet  of  this  method  is  that  the  designed  plan  will  not  be  perfect,  without  error  or 
always  get  it  right,  especially  in  its  initial  inception.  The  goal  of  a  plan’s  first  iteration 
is  to  provide  a  baseline  for  results  assessment  that  the  staff  will  then  use  to  improve  and 
update  the  plan.  It  is  vital  that  the  staff  completes  the  evaluation  of  the  entire  plan  in 
congruence  with  achievement  of  effects  and  outcomes  (both  desired  and  undesired). 
Williams  and  Morris  warn  against  using  assessment  just  to  review  and  modify  the  chosen 
course  of  action.  Instead,  they  note,  it  is  important  to  evaluate  all  conditions,  not  just 
those  objectives  and  end-states  achieved,  and  use  them  to  create  entirely  new  plans  based 
on  dynamic  circumstances.  Evaluation  theory  clearly  states  that  assessment  requires 
focusing  not  only  on  assuring  that  the  plan  gets  accomplished,  but  constantly  reviewing 

Douglas  J.  MacIntyre,  “Operational  Assessment:  Replacing  Black  Magic  with  Learning  by 
Design,”  Marine  Corps  Gazette  95,  no.  6  (June  2011),  20. 

Williams  and  Morris,  75. 
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the  plan  to  ensure  that  it  is  achieving  the  correct  and  most  current  goals. 

For  the  military  to  adopt  the  evaluation  concept,  it  will  need  to  create  a 
methodology  that  focuses  on  the  theory  of  change  espoused  by  the  plan.  David  Apgar, 
when  writing  about  assumption-based  metrics,  a  theory  comparable  to  theory-driven 
evaluation,  notes  that  the  starting  point  of  the  process  is  for  executives  to  use  their 
intuition  to  determine  what  they  consider  most  influential  or  important.  Then,  the 
company  uses  evaluation  to  test  that  theory  and  see  if  it  is  correct  or  needs  adjustment. 

He  goes  on  to  explain  that  assessment  metrics  are  much  like  recipe  ingredients;  alone  and 
unused,  they  are  not  of  much  relevance.  But,  the  cook  (or  commander)  selects  certain 
ingredients  and  the  amount  of  each  to  use  by  developing  a  recipe  that  puts  them  together 
and  hopefully  produces  a  culinary  masterpiece.  The  only  way  to  see  if  the  outcome  is 

OA 

correct  is  to  execute  the  recipe  and  analyze  the  resulting  product.  In  the  same  way,  the 
elements  of  operational  design  are  the  main  ingredients  of  the  plan.  The  commander 
develops  a  theory  on  how  the  combination  of  these  ingredients  will  lead  to  acceptable 
outcomes.  It  is  only  by  testing  these  outcomes  to  see  if  they  meet  the  current  political 
interests  of  the  nation  that  it  is  possible  to  validate  the  theories.  The  most  important  part 
of  evaluation  is  to  compare  the  current  outcomes  with  the  theories  that  are  the  basis  for 
the  plan  and  then  assess  if  the  realized  outcomes  validate  those  theories. 

Reviewing  and  Improving  the  Assessment  Plan 
Recent  joint  doctrine  has  done  a  good  job  tying  the  creation  of  an  assessment  plan 
or  development  of  metrics  to  the  actual  planning  process.  The  U.S.  assessment  handbook 

Apgar,  David,  “Assumption-Based  Metrics:  Recipe  for  Success,”  Strategic  Finance  93,  no.  5 
(November  201 1),  28. 

Ibid.,  28  -  29. 
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affirms  that  the  first  step  of  assessment  is  to  support  the  creation  of  the  operational  plan 

o  1 

while  developing  the  assessment  plan  in  tandem.  It  stresses  that  assessment  members 
must  be  a  part  of  the  selection  of  end-states,  objectives,  and  effects  to  ensure  they  are 
written  in  a  manner  that  is  measurable.  These  statements  allow  the  assessment  team 
involvement  in  the  creation  of  the  operational  plan  and  helps  capture  key  decisions  and 
linkages  that  are  determined  during  the  development  of  the  operational  approach. 

Further,  it  allows  them  to  develop  an  assessment  plan  that  closely  links  and  coordinates 
with  the  planned  actions  to  ensure  the  most  accurate  assessment  and  understanding  of 
what  the  plan  needs  to  accomplish. 

The  problem  with  this  doctrine  for  the  development  of  the  assessment  plan  is  that 
it  focuses  almost  entirely  on  the  creation  of  the  assessment  plan  during  plan  development. 
The  assessment  handbook  beautifully  lays  out  the  six  steps  for  developing  a  successful 
assessment  plan.  Unfortunately,  it  ends  at  the  planning  stage  of  the  operation  and 
mentions  no  methodology  for  reviewing  or  updating  the  assessment  plan  during 
execution.  If  the  staff  does  not  regularly  update  the  assessment  as  the  campaign 
progresses,  it  loses  touch  with  the  changing  operational  environment  and  no  longer 
provides  meaningful  information.  Metrics  have  a  shelf-life  and  need  adjustment  as  the 
strategy  and  execution  of  the  plan  adapt  to  changing  situations. This  means  that 
commander  and  staff  must  evaluate  and  update  the  assessment  plan  to  ensure  it  provides 
the  correct  information  they  need  to  properly  assess  the  operational  plan  in  order  to  make 

U.S.  Joint  Staff,  J-7,  Commander’s  Handbook  for  Assessment  Planning  and  Execution,  VI-1. 

Ibid. 

Ibid.,  IV-3. 

Robert  B.  Sotire,  Measuring  Performance  and  Effectiveness  in  Irregular  Warfare:  Preventing 
Dysfunctional  Behavior,  (Newport,  RI:  Naval  War  College,  2009), 
http://handle.dtic.mi1/100.2/ADA504933  (accessed  August  12,  2013),  13. 
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the  necessary  changes. 

Monitoring  Assumptions 

An  important  part  of  the  Joint  Operations  Planning  Process  during  the  Mission 
Analysis  step  is  the  initial  identification  of  assumptions.  This  is  a  crucial  procedure 
because,  in  complex  and  dynamic  operational  environments,  it  is  rare  to  develop  a  plan 
without  making  basic  assumptions  to  help  narrow  the  scope  of  the  plan  and  simplify  the 
options.  History  is  replete  with  examples  of  plans  that  failed  because  their  underlying 
assumptions  proved  to  be  false.  The  most  recent  U.S.  example  of  this  reality  is  the 
failure  of  Operation  IRAQI  FREEDOM  to  have  any  significant  Phase  IV  planning  to 
handle  controlling  Iraq  after  accomplishing  the  desired  regime  change.  In  his  book  The 
Generals,  Thomas  Ricks  highlights  that  both  President  Bush’s  staff  and  the  CENTCOM 
commander.  General  Tommy  Eranks,  assumed  that  post- invasion  Iraq  would  welcome 
U.S.  forces  with  open  arms  and  embrace  the  freedom  they  had  been  given.  General 
Eranks  fully  expected  and  briefed  the  White  House  that  the  U.S.  forces  would  be 
completely  out  of  Iraq  the  same  year  as  the  invasion  occurred.  This  was  a  key 
assumption  and  laid  the  groundwork  for  all  Phase  IV  planning  and  tasks  that  the  U.S. 
military  accomplished.  Unfortunately,  it  was  never  well  monitored  or  evaluated  during 
execution  and  it  took  several  years,  during  which  an  insurgency  and  civil  war  developed 
and  overtook  the  country,  before  national  and  CENTCOM  leaders  properly  accepted  it  as 
a  false  assumption. 

While  staffs  often  spend  significant  planning  effort  debating  the  assumptions 


^  Thomas  E.  Ricks,  The  Generals:  American  Military  Command  From  World  War  II  to  Today, 
(New  York,  Penguin  Press,  2012),  403. 
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involved  in  an  operation,  it  is  easy  for  these  assumptions  to  get  lost  in  the  joint  operations 
planning  process  as  it  shifts  to  COA  development,  selection,  order  writing  and  execution, 
to  the  point  where  many  people  executing  the  plan  will  have  no  idea  of  the  assumptions 
upon  which  it  is  based.  Joint  doctrine  is  aware  of  the  importance  of  assumptions  and  the 
effect  that  their  confirmation  or  rejection  can  have  on  the  outcome  of  an  operation. 

on 

JP  5-0  stresses  that  “assumptions  must  be  continually  reviewed  to  ensure  validity.” 

But,  joint  doctrine  is  currently  very  plan-focused  and  continues  that  “although  there  may 
be  exceptions,  the  staff  should  strive  to  resolve  all  assumptions  before  issuing  the 

-5  0 

OPORD.”  This  expectation  is  short-sighted  and  almost  impossible  to  fulfill.  Most 
staffs  base  assumptions  on  the  way  adversaries,  neutrals,  and  friendly  or  coalition  forces 
will  act  or  respond  during  the  execution  of  the  plan.  It  is  not  possible  to  validate  many 
assumptions  before  executing  the  plan  and  the  enemy  response  becomes  clear.  The  U.S. 
assessment  handbook  does  make  several  assertions  that  the  assessment  teams  need  to 
occasionally  validate  assumptions  and  that  they  should  continuously  challenge 
assumptions  during  evaluation.  It  even  mentions  that  the  mechanism  used  to  conduct 
the  assumption  validation  is  MOEs.'^^  While  these  may  be  good,  general  statements,  what 
is  lacking  is  any  information  on  how  to  accomplish  the  validation  process  or  even  how  to 
develop  MOEs  for  monitoring  and  evaluating  assumptions.  Since  MOEs  focus  on 
evaluating  effects  and  not  facts,  they  are  completely  the  wrong  mechanism  for  validating 
the  assumptions  that  are  the  underlying  basis  of  the  plan.  While  the  importance  of 
assumptions  and  the  need  to  monitor  them  is  accepted  joint  doctrine,  there  is  no 

U.S.  Joint  Chiefs  of  Staff,  Joint  Operation  Planning,  IV-8. 

Ibid. 

U.S.  Joint  Staff,  J-7,  Commander’s  Handbook  for  Assessment  Planning  and  Execution,  II-6. 

Ibid.,  IV-5. 
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established  or  even  practiced  mechanism  for  conducting  this  important  requirement. 

To  consider  possible  solutions  to  this  problem,  it  is  best  to  re-consider  the  civilian 
practice  of  Theory  of  Change  strategies  to  see  how  they  address  this  issue.  A  key 
component  of  the  Theory  of  Change  is  the  identification  of  the  assumptions  that  form  the 
basis  for  the  theory  The  staff  does  not  accept  the  listed  assumptions  as  facts  until 
proven.  Instead  they  closely  monitor  each  of  the  assumptions  throughout  execution,  and 
where  possible,  develop  tests  to  confirm  or  deny  the  validity  of  the  assumptions  as  soon 
as  reasonably  possible.  Overall,  joint  doctrine  identifies  assumptions  as  a  key  part  of 
the  joint  planning  process.  What  it  lacks  is  a  clear  assessment  method  for  the  assessment 
team  to  use  in  monitoring  and  evaluating  these  critical  elements  as  the  plan  and  execution 
of  the  operation  proceeds. 


Treasury  Board  of  Canada  Secretariat,  Theory-Based  Approaches  to  Evaluation:  Concepts  and 
Practices,  (Ottawa,  Canada:  Treasury  Board  of  Canada  Secretariat,  2012),  5. 

Dana  H.  Taplin  and  Helene  Clark,  Theory  of  Change  Basics:  A  Primer  on  Theory  of  Change, 
(New  York:  ActKnowledge,  2012), 
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CHAPTER  4:  ASSESSMENT  FRAMEWORK  IS  NOT  ADEQUATE 

You  know  what  is  wrong  with  a  lot  more  confidence  than  you  know  what  is 
right. 

-  Nassim  Nicholas  Taleb,  The  Black  Swan:  The  Impact  of  the  Highly  Improbable 
Current  Assessment  Doctrine  is  Inadequate 
The  current  state  of  U.S.  military  doctrine  on  assessment  does  not  provide 
practitioners  enough  information  to  properly  execute  assessment  methodology. 

Numerous  professional  journal  and  periodical  articles  written  by  assessment  experts  over 
the  last  decade  decrying  the  current  state  of  U.S.  military  assessment  practices  document 
this  fact  well.  Probably  the  biggest  critic  of  current  assessment  doctrine  is  Jonathan 
Schroden,  former  professor  at  the  Naval  War  College,  who  bluntly  states,  “One  reason 
operations  assessments  fall  short  is  that  there  are  deficiencies,  contradictions,  and 
confusion  in  the  doctrine  that  is  supposed  to  guide  their  conduct.”^  He  goes  on  to 
conclude  that  current  assessment  problems  run  much  deeper  than  just  poor  metrics  and 
faulty  mathematics,  saying  that  what  is  required  is  a  complete  rethink  of  the  theory 
behind  assessment  plans  and  execution.  It  is  obvious  that  the  current  state  of  assessment 
doctrine  is  the  root  cause  behind  the  poor  execution  of  assessment  by  military  staffs. 

Based  on  the  rash  of  literature  available  on  this  subject  and  recent  publications,  it 
seems  that  the  military  is  aware  of  the  deficiency  and  making  some  efforts  toward 
rectifying  it.  The  release  of  the  U.S.  Assessment  Handbook  in  201 1  and  the  NATO 
Assessment  Handbook  in  2012  finally  provided  some  guidance  on  various  methodologies 
assessment  teams  can  use  to  develop  and  execute  assessment  plans.  The  U.S.  followed 

*  Jonathan  Schroden,  "Why  Operations  Assessment  Fails:  It's  Not  just  the  Metrics,"  Naval  War 
College  Review  64,  no.  4  (Autumn  2011),  92. 

^  Ibid.,  96-99. 
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up  their  assessment  document  with  the  July  2013  release  of  an  Insights  and  Best  Practices 
Focus  Paper  on  Assessment.  This  paper  is  a  useful  review  of  current  assessment 
practices  used  by  various  Combatant  Commanders  in  operational  theaters  with  a 
highlight  of  best  practices  seen  in  each  location.  Unfortunately,  neither  the  U.S.  nor 
NATO  seems  to  be  engaged  in  taking  the  next  step  of  putting  these  best  practices  or 
methodologies  into  the  extensive  formal  doctrine  statements  required  to  standardize 
practices  across  commands.  Walter  Ledford,  a  doctrine  writer  in  the  Joint  Staff  J-7, 
explained  in  an  interview  that  the  U.S.  Joint  Staff  is  conducting  an  assessment  survey  of 
Combatant  Command  staffs  to  try  to  determine  the  best  way  forward  for  U.S.  military 
assessment  development  and  will  be  publishing  a  short  Joint  Doctrine  note  on  assessment 
sometime  in  2014."^  For  now,  the  joint  community  is  not  preparing  for  or  willing  to 
codify  extensive  changes  to  assessment  practices  in  doctrine. 

Lack  of  Defined  Assessment  Organizational  Structure 
The  military  has  conducted  assessment  for  long  enough  to  amass  a  large  body  of 
journal  articles,  commentaries,  and  joint  best  practices  on  the  subject.  Nearly  all  of  them 
come  to  the  same  conclusion:  assessment  is  a  field  easily  misused  and  abused  if  not 
conducted  by  professionals  educated  in  proper  statistical  methods  and  mathematical 
fallacy  avoidance.  Unfortunately,  military  assessment  teams  (AT)  rarely  contain  experts 
who  have  the  required  professional  level  of  training.  This  is  contradictory  to  the  basic 
concepts  of  the  resource-based  management  theory.  Under  this  theory,  it  is  vital  for  the 
evaluators  to  have  the  technical  and  methodological  expertise  required  by  the  program 


^  U.S.  Joint  Staff,  J-7,  Deployable  Training  Division,  Insights  and  Best  Practices  Focus  Paper: 
Assessment,  (Washington  DC;  Joint  Staff,  J-7,  Deployable  Training  Division,  July  2013),  1. 

Walter  Ledford,  Joint  Staff  J-7,  interview  by  author,  January  8,  2014. 
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theory  as  well  as  understand  the  program  itself.^  Some  parts  of  the  military  seem  to 
recognize  this  void.  U.S.  Army  Civil  Affairs  and  Psychological  Operations  Command 
guidance  dictates  the  use  of  reachback  capability  during  assessment  metric  evaluation  in 
order  to  access  statistical  expertise  rarely  available  in  theater.  NATO  operations 
assessment  doctrine  strikes  a  similar  note  when  it  states  that  a  multi-discipline  team  must 
conduct  assessment,  constituted  by  systems  analysts,  social  scientists  and  operational 
analysts,  each  of  which  has  a  unique  skill  set  for  which  there  is  no  substitute.^  In  order  to 
work  correctly,  commands  must  staff  assessment  teams  (AT)  with  properly  skilled 
individuals. 

The  military  is  doing  a  poor  job  of  developing  ATs  that  have  the  skill  sets 
required  to  create  relevant  and  correct  assessments.  Jonathan  Schroden,  who  has  visited 
assessment  teams  in  multiple  combat  operations,  points  out  that  in  his  experience,  ATs 

~j 

are  staffed  with  unqualified  officers  or  untrained  operational  researchers.  The  root  of  the 
problem  lies  in  the  fact  that  military  instructions  set  no  expectations  or  requirements  for 

o 

assessment  teams  (ATs).  Without  a  doctrinal  basis  or  requirement,  especially  in  a 
fiscally  constrained  environment,  joint  staffs  cannot  justify  the  need  for  assessment 
experts  in  their  ATs.  Exacerbating  the  problem  is  the  fact  that  current  ATs  are  filled  and 
led  by  military  officers  who  are  trying  to  accomplish  the  job  without  any  formal  training 


^  Andrew  P.  Williams  and  John  C.  Morris,  “The  Development  of  Theory-Driven  Evaluation  in  the 
Military:  Theory  on  the  Front  Line,”  American  Journal  of  Evaluation  30,  no.  1  (March  2009),  65. 

®  NATO  Operational  Experiment,  NATO  Operations  Assessment  Experiment  Final  Report 
(Norfolk,  VA:  Headquarters,  Supreme  Allied  Commander  Transformation,  2013),  17. 
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or  background  in  assessment.^  These  officers  often  read  the  meagre  doctrine  that  exists 
on  assessment,  but  find  no  real  information  to  help  them  in  their  job  performance. 

The  creation  of  the  U.S.  Joint  assessment  handbook  published  in  201 1  has  finally 
given  some  general  advice  on  the  creation  and  execution  of  an  assessment  team  (AT).^*^ 
While  it  is  a  good  start,  it  lacks  many  of  the  elements  required  to  provide  ATs  with  the 
expertise  they  need  to  accomplish  the  job.  First,  the  nature  of  the  handbook  is  such  that  it 
is  not  doctrinal  guidance  like  joint  publications,  but  rather  “it  provides  fundamental 
principles,  techniques,  and  considerations  related  to  assessments  that  are  being  employed 
in  the  field.”^'  This  means  that  the  handbook  only  contains  suggestions  and  best 
practices  that  are  still  under  development  so  they  do  not  provide  staffs  with  a  basic 
methodology  or  procedure  like  doctrine.  Next,  the  handbook  is  very  vague  on  the  best 
practice  solutions  for  the  creation  of  assessment  working  groups  or  ATs.  The  handbook 
acknowledges  that  each  command  currently  builds  its  ATs  differently  with  diverse  make¬ 
ups  of  assessment  trained  personnel,  subject  matter  experts,  and  group  composition.^^  By 
providing  no  baseline  solution,  there  is  no  basis  for  commands  to  work  toward  a  common 
solution  or  push  for  funding  of  expertise  positions  on  their  staffs.  Finally,  the  handbook 
is  also  inconclusive  on  the  lead  directorate  placement  for  the  AT;  whether  it  best  belongs 
under  the  J-2,  J-3,  or  J-5  directorate.'^  The  Joint  Staff  J-7  recently  issued  clear  guidance 
in  the  document  Assessment  Best  Practices  where  they  concluded,  “At  the  Operational 

^  Sebastiann  Rietjens,  Joseph  Soeters,  and  Willem  Klumper,  “Measuring  the  Immeasurable?  The 
Effects-Based  Approach  in  Comprehensive  Peace  Operations,”  International  Journal  of  Public 
Administration  34,  no.  5  (2011),  333. 
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Level,  consider  establishment  of  an  assessment  cell  in  the  J-5  or  as  an  empowered 
separate  directorate.”'"^  Until  the  joint  staff  J-7  establishes  a  doctrinal  baseline  for  ATs 
and  professional  requirements,  joint  staffs  will  continue  to  struggle  to  create  and  fill  these 
positions  with  personnel  that  can  accomplish  assessment  correctly. 

Absence  of  Formal  Terminology  and  Training 
A  complete  lack  of  standard  vocabulary  or  terminology  also  plagues  current 
assessment  methodology.  This  deficiency  first  appears  in  the  planning  arena  where 
assessment  teams  use  terms  such  as  operational  objective,  tactical  objectives,  and  tactical 
tasks,  but  U.S.  military  doctrine  does  not  define  these  terms. Currently,  there  is  also  no 
formal  U.S.  assessment  training.  This  means  military  personnel  recruited  into  the  ATs 
must  conduct  assessments  while  learning  on  the  job,  learning  from  their  predecessors,  or 
by  reading  the  paltry  doctrine  that  exists  on  the  subject.'^  The  problem  with  any  of  these 
methods,  as  discussed  in  the  previous  section,  is  that  assessment  is  a  technical  field  that 
requires  expertise;  otherwise  the  assessor  can  easily  fall  into  any  of  a  number  of 
mathematical  fallacies  or  logical  pitfalls  in  their  evaluations.  The  fact  that  the  recent  war 
zones  of  Iraq  and  Afghanistan  create  most  assessment  knowledge  further  exacerbates  the 
problem.  This  is  a  problem  because  the  military  tends  to  assign  assessment  to  their 
positions  for  six  months  to  one  year,  with  most  of  the  staff  rotating  at  the  same  time.  As 
a  result,  each  rotation  tends  to  develop  a  tailored  assessment  system,  not  understanding  or 
building  upon  the  products  previously  developed  and  having  to  relearn  hard  taught 

U.S.  Joint  Staff,  J-7,  Deployable  Training  Division,  10. 

Stephen  Downes -Martin,  "Operations  Assessment  in  Afghanistan  is  Broken;  What  is  to  be 
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lessons.  Even  the  U.S.  assessment  handbook  recognizes  the  desperate  need  for 
standardized  terms  but  must  state  that  it  is  neither  authoritative  nor  prescriptive,  and  that 
it  hopes  a  common  set  of  terms  and  processes  will  ultimately  ensue.  Until  that  time 
comes,  the  lack  of  standardized  terms  and  training  is  significantly  impairing  the 
capability  of  assessing  military  operations. 

As  much  as  there  is  a  lack  of  common  terminology  in  military  assessment,  there  is 
also  a  severe  paucity  of  formal  training  and  definition  for  assessment  teams  (ATs). 
Jonathan  Schroden  writes  that  a  critical  problem  with  the  current  practice  of  assessment 
is  that  there  is  no  training  pipeline  for  practitioners  and  there  is  no  dedicated  cadre  of 
experts  being  built  up  to  improve  the  field. These  two  problems  are  mutually 
destructive  because  the  military  is  currently  building  up  a  large  volume  of  assessment 
experience,  knowledge,  and  lessons.  Although  there  is  a  formal  process  to  capture  this 
information,  the  military  does  not  return  it  to  the  field  by  training  the  next  wave  of 
assessment  teams. 

The  U.S.  military  is  not  without  a  monitoring  and  feedback  system,  and  the  Joint 
Lessons  Learned  team  has  collected  data  on  the  assessment  system  and  provided 
feedback  to  the  Joint  Staff  J-7  for  required  improvements.  This  feedback  is  what  led  to 
products  like  the  U.S.  assessment  handbook  and  assessment  focus  paper.  Unfortunately, 
the  efforts  of  the  lessons  learned  team  do  not  go  nearly  far  enough.  The  military  does  not 
staff  the  team  with  assessment  experts  and,  while  they  can  collect  small  improvements 
and  recommended  changes,  they  are  not  a  good  organization  for  creating  significant 

NATO  Operations  Assessment  Sub-Working  Group,  NATO  Operations  Assessment  Handbook, 
(Norfolk,  VA:  NATO  Supreme  Allied  Commander  Transformation,  2012),  2-2. 

**  U.S.  Joint  Staff,  J-7,  Commander’s  Handbook  for  Assessment  Planning  and  Execution,  1-7. 
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change  or  overhauling  the  current  system.  A  significant  level  of  change  is  required  in  the 
assessment  area  and  only  a  dedicated  group  of  assessment  subject  matter  experts  can  do  it 
well.  Also,  the  lessons  learned  team  provides  feedback  to  personnel  already  conducting 
assessment  and  does  not  provide  the  baseline  training  that  is  necessary  to  develop  a 
competent  staff.  Without  emphasizing  the  improvement  of  assessment  doctrine  and 
creating  baseline  standards  and  training  for  assessment  practitioners,  the  U.S.  military 
will  not  be  able  to  make  necessary  improvements  in  assessment  methodology. 

Assessment  Deficiency  in  Sharing  Information  between  Organizations 

One  other  area  where  there  is  a  general  dearth  of  information  on  assessment 
methodology  is  the  establishment  and  sharing  of  assessment  products  between  various 
military  organization  levels.  JP  5-0  broaches  the  subject  and  makes  broad  assertions 
about  the  need  to  link  the  organization  levels  and  that  the  organization  level 
accomplishing  the  task  should  also  be  the  one  responsible  for  conducting  the  assessment 
of  that  area.  The  U.S.  assessment  handbook  does  not  provide  much  more  detail  when  it 
asserts  that  assessment  teams  need  to  ensure  that  all  levels  share  assessment  products, 
remain  well  connected,  and  properly  link  supported/supporting  plans. These  are  good 
statements  that  frame  the  goal  of  assessment  across  multiple  military  echelons,  but  both 
documents  are  lacking  any  discussion  of  how  to  link  the  products,  connect  the  various 
assessment  plans  or  share  information. 

The  reality  of  accomplishing  these  seemingly  simple  principles  is  what  current 
assessment  practitioners  are  finding  difficult.  U.S.  Marine  Lieutenant  Colonel  Douglas 
MacIntyre,  who  returned  from  operations  in  Afghanistan  in  2010,  wrote  that  ISAF 

U.S.  Joint  Chiefs  of  Staff,  Joint  Operation  Planning,  D-7. 

U.S.  Joint  Staff,  J-7,  Commander’s  Handbook  for  Assessment  Planning  and  Execution,  I-IO. 
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developed  an  overall  assessment  strategy,  but  the  regional  commanders  did  not  trust  or 
agree  with  it,  so  each  developed  their  own  assessment  methodology  and  reporting 
process.  In  the  same  manner,  U.S.  Air  Force  Lieutenant  Colonel  Kristen  Messer  wrote 
that,  in  her  experience.  Joint  Task  Force  staffs  tend  to  reserve  the  operational  evaluation 
for  themselves  and  disregard  evaluation  information  sent  up  by  their  subordinate 
commands.  This  leads  them  to  neglect  the  subordinate  commander’s  evaluation 
because  they  are  asking  for  only  raw  monitoring  data  and  making  the  assessments 
themselves.  Coalition  operations  complicate  the  problem  even  further  because  each 
nation  wants  to  develop  and  conduct  its  own  assessment  plan  and  methodology.  This  is 
exactly  what  is  happening  in  Afghanistan  where  ISAF,  the  U.S.,  the  UK,  Holland,  and 
Canada  each  have  independent  evaluation  systems.  One  other  problem  with  current 
assessment  sharing  between  levels  is  that  subordinate  levels  rarely  received  feedback  on 
the  superior  organization’s  evaluations  and  use  of  subordinate  level  data.  This  means 
that  they  are  collecting  and  analyzing  data  without  knowing  if  it  is  fulfilling  its  purpose 
or  if  the  superior  organization  considers  their  evaluations  correct  or  accurate.  Since  there 
is  no  unity  of  assessment  methodology  between  levels  and  outside  organizations,  the 
hope  of  building  a  comprehensively  informed  and  holistic  view  of  the  campaign  is  almost 
impossible. 

Inconsistency  of  Assessment  Output  Products 

The  final  procedural  problem  with  assessment  is  that  it  inappropriately  relies  on 


Douglas  J.  MacIntyre,  “Operational  Assessment:  Replacing  Black  Magic  with  Learning  by 
Design,”  Marine  Corps  Gazette  95,  no.  6  (June  2011),  19. 

Kristen  Messer  and  Shane  Dougherty,  “A  New  Operational  Assessment  Paradigm:  Splitting  the 
Stoplights,”  Air  &  Space  Power  Journal  20,  no.  3  (Fall  2006),  59. 
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simplistic  quantitative  procedures.  The  current  output  method  of  military  assessment 


products  is  almost  completely  quantitative  in  nature  and  uses  the  unsophisticated  and 
often  misleading  form  of  stoplight  charts  and  quantitative  bar  or  line  graphs.  The 
examples  given  in  the  U.S.  military  assessment  handbook  are  entirely  stoplight  charts  and 
have  no  associated  narratives  or  textual  information  provided  with  the  evaluations.^^  The 
same  is  true  for  the  sample  products  in  the  assessment  focus  paper,  which  provides  two 
examples  that  are  either  stoplight  charts  or  sliding  scales  with  no  associated  textual 
information.  These  quantitative  types  of  charts  can  be  very  appealing  to  staffs  and 
commanders  because  they  are  simple  and  provide  what  appear  to  be  very  concrete 
evaluations.  If  the  staff  rates  the  areas  green  or  moving  to  the  positive  side,  then  the 
military  is  winning  and  operations  are  going  well.  The  problem  with  these  charts  is  they 
dramatically  oversimplify  the  assessments  provided  and  contradict  the  fundamental  tenets 
of  the  theory  that  assessment  is  based  upon.  Stoplight  charts  are  great  for  tracking  plan 
execution  and  answering  “If  we  are  doing  things  right?”  But,  they  cannot  be  used  to 
explain  causality,  influence,  or  evaluate  the  correctness  of  the  selected  elements  of 
operational  design.  Evaluations  of  these  elements  require  a  narrative  that  describes 
causes,  suspected  interactions,  and  intended  and  unintended  consequences. 

no 

An  alternate  method  of  output  display  is  the  integration  of  a  structured  narrative 


U.S.  Joint  Staff,  J-7,  Commander’s  Handbook  for  Assessment  Planning  and  Execution,  V-9. 

U.S.  Joint  Staff,  J-7,  Deployable  Training  Division,  11. 

A  Structured  Narrative  is  a  compromise  format  between  free  text  and  quantitative  charts. 
Structured  Narratives  consist  of  text  that  follow  an  explicit  format  and  specified  set  of  fields  and  text  with 
words  that  are  selected  based  on  their  relevence  to  the  subject  area  as  well  as  spectrum  of  descriptive 
ability.  The  Structured  Narrative  allows  an  assessor  to  provide  relevance  and  context  to  the  quantitative 
information  that  is  provided  while  using  a  format  that  allows  higher  organizations  to  review  and  combine 
multiple  narratives  in  a  cogent  manner.  There  are  even  computer  programs  designed  to  collect  Structured 
Narrative  inputs,  combine  them  and  produce  consolidated  Structured  Narrative  output,  just  like  can  be  done 
with  quantitative  data.  See  Appendix  A  for  Structured  Narrative  examples. 
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into  the  stoplight  and  sliding  metric  charts  that  currently  exist.  The  key  missing 
capability  of  a  sliding  chart  is  preservation  of  the  extreme  but  important  data  that  the  staff 
can  easily  ignore  or  become  overwhelmed  by  when  aggregating  quantitative  data.  For 
example,  if  the  staff  aggregates  the  current  statuses  of  a  country  comprised  of  100  regions 
to  give  an  overall  assessment,  it  is  easy  to  add  99  regions  that  are  doing  well  to  a  single 
region  that  is  doing  poorly  and  assess  the  overall  country  as  green.  That  result  can  hide  a 
single  poor  area  that  has  a  population  larger  than  most  of  the  other  areas  combined  or  that 
is  the  source  of  the  majority  of  drugs  or  insurgents  in  the  region.  Adding  a  structured 
narrative  to  the  stoplight  chart  helps  preserve  the  needed  contextual  detail  of  the 
assessment. 

The  use  of  structured  narrative  is  a  contentious  topic  in  the  assessment  community 
and  there  is  an  on-going  debate  in  assessment  literature  about  the  value  and  integration  of 
structured  narrative.  Current  military  opinion  sides  against  it  based  on  the  fact  that  the 
current  military  assessment  handbooks  do  not  mention  it  nor  show  it  in  the  examples.  In 
contrast,  there  are  civilian  assessment  experts  like  Stephen  Downes-Martin,  research 
professor  at  the  Naval  War  College,  who  recommend  the  assessment  of  metrics  using 
exclusively  narratives. 

Structured  narrative  methodology  does  have  some  obvious  drawbacks.  The  first 
is  the  fact  that  it  can  lead  to  information  overload.  A  stoplight  chart  can  provide  a  large 
amount  of  information  on  a  slide  that  facilitates  easy  viewing  and  understanding.  That 
same  information  in  text  form  could  require  several  slides  or  the  staff  can  place  it  on  a 

National  Defense  Research  Institute,  Improving  Counterinsurgency  Campaign  Assessment:  The 
Importance  of  Transparency  in  the  Fog  of  War,  (Washington  DC;  RAND  Corporation,  2012),  2. 
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single  slide  in  an  illegible  small  font.  ISAF  tested  the  use  of  narratives  on  their 
assessment  slides,  but  the  British  Colonel  who  chaired  the  assessment  group  rejected  the 

O  1 

method  because  he  felt  that  it  was  putting  too  much  information  on  a  single  slide. 

Analysts  also  tend  to  struggle  with  free  narrative  because  it  is  difficult  to  combine  and 
determining  what  is  important  enough  to  retain  and  what  is  unnecessary  when 
aggregating  information  is  very  subjective.  In  Afghanistan,  ISAF  produced  so  much 
unstructured  narrative  data  that  it  inundated  the  system  and  prevented  proper  assessment 
of  the  environment.  The  creation  of  structured  narratives  occurred  to  overcome  these 
two  criticisms  because  they  are  limited  in  length  by  a  specified  format  and  contain  a 
structure  that  allows  computer  programs  to  combine  them  to  provide  an  overall 
assessment.  Still,  the  use  of  narrative  alone  does  not  fit  the  needs  of  the  Combatant 
Commander’s  decision  makers. 

The  best  approach  is  a  balanced  combination  of  quantitative  data  and  structured 
narratives.  This  method  preserves  the  advantages  of  both  systems.  The  advantages  of  the 
summary  graphics  are  that  they  are  easily  produced  and  understood,  and  tend  to  give  a 
macro-level  view  of  the  current  campaign  progress.  In  contrast,  structured  narratives 
allow  for  the  preservation  of  the  important  context  of  the  data  as  well  as  a  providing 
qualitative  assessment  of  the  data.^^  The  proper  balancing  of  these  two  processes  into  a 
single  methodology  is  necessary  to  provide  the  assessment  community  with  the  best  set 
of  tools  to  inform  the  commander’s  decision-making  and  avoid  the  pitfalls  of  misleading 

Rietjens,  Soeters,  and  Klumper,  335. 
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evaluations.  It  is  only  through  the  inclusion  of  the  structured  narrative  to  complement  the 
summary  graphics  that  an  assessment  team  can  impart  the  key  context  and  outlying  data 
that  is  necessary  to  provide  the  commander  with  the  best  assessment  information. 

Unique  Requirements  of  Traditional  Warfare  Assessment 
While  assessment  is  currently  a  topic  of  heated  debate  and  discussion  in  the 
military  and  operations  research  communities,  most  of  the  recent  dialogue  focuses  on  the 
use  of  assessment  in  counterinsurgency  campaigns  that  tend  to  be  longer-term,  slower- 
paced  operations.  The  military  normally  conducts  these  operations  over  long  periods  of 
time  with  multiple  transitions  of  commanders  and  staffs,  and  plenty  of  time  to  gather, 
analyze,  evaluate,  and  act  upon  assessment  data.  Rarely  discussed  in  current  literature  is 
the  methodology  for  conducting  assessment  in  the  fast-paced  and  quickly  shifting 
operations  that  tend  to  characterize  traditional  conventional  warfare.  As  doctrine  revision 
occurs,  it  is  important  to  address  the  unique  elements  of  assessment  in  this  type  of 
warfare  and  layout  techniques,  most  of  which  will  be  different  from  those  used  during 
irregular  warfare  such  as  counter-insurgency  or  stability  operations. 

The  current  perception  of  conducting  assessment  during  traditional  warfare  by  the 
assessment  community  is  that  it  is  easier  and  fairly  straightforward.  The  community 
holds  this  belief  because  traditional  warfare  tends  to  have  more  clearly  defined 
objectives;  such  as  taking  specific  terrain,  destroying  a  certain  part  of  the  enemy 
infrastructure  or  defeating  an  element  of  the  enemy  military.^®  From  this  viewpoint, 
traditional  warfare  assessment  is  more  simplistic  because  the  metrics  are  easy  to  define 

The  definition  and  examples  of  irregular  warfare  is  taken  from  JP  5-0,  III-43. 

Jason  Campbell,  Michael  E.  O'Hanlon,  and  Jeremy  Shapiro,  "How  to  Measure  the  War:  Judging 
Success  and  Failure  in  Counterinsurgency,"  Policy  Review,  no.  157  (October  2009),  15. 
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and  collect  against.  Joint  doctrine  lays  out  an  entire  process  called  Combat  Assessment 
that  is  well  suited  to  measuring  tactical  level  tasks  and  effects  in  a  traditional  force-on- 
force  conflict.  From  a  monitoring  standpoint,  current  doctrine  is  well-suited  to 
traditional  warfare  assessment. 

There  are  two  areas  where  doctrine  is  lacking  for  traditional  warfare  assessment: 
dealing  with  the  high-tempo  of  operations,  and  evaluating  the  effects  and  outcome  of  the 
campaign.  The  U.S.  assessment  handbook  makes  only  one  mention  of  high-tempo 

-50 

operations,  indicating  that  the  assessment  cycle  may  need  to  operate  more  frequently. 
This  is  a  rather  gross  understatement  of  the  problem.  Messer  points  out  that  the  current 
manning  or  equipping  tactical-level  assessment  teams  does  not  enable  them  to 
accomplish  the  large  amount  of  information  and  combat  assessments  that  need  to  take 

-50 

place  during  traditional  warfare.  If  the  tactical  level  cannot  generate  the  lower  level  of 
inputs  into  the  system,  then  the  joint  level  will  not  have  the  current  and  complete 
information  it  needs  to  make  its  higher- level  assessments. 

The  second  problem  is  that  the  staff  must  analyze  the  data  and  provide 
recommendations  to  the  commander  in  a  timely  manner.  The  problem  with  high-paced 
operations  is  that  they  tend  to  overwhelm  the  assessment  team’s  time  with  just  collecting 
and  sorting  the  data  that  is  flooding  in.  It  leaves  them  with  no  capacity  to  develop  an 
overall  assessment  of  the  operation  progression.  In  these  situations,  the  tendency  is  for 
the  assessment  staff  to  move  closer  to  the  J-3  directorate  because  the  J-3  is  looking  for 
immediate  feedback  on  its  operations  so  it  can  retarget  or  retask  unsatisfactorily 

U.S.  Joint  Chiefs  of  Staff,  Joint  Operation  Planning,  D-7. 
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accomplished  areas. While  this  tactical  level  focus  is  important,  it  can  lead  the 
assessment  team  to  fall  victim  to  the  old  adage  of  losing  the  forest  for  the  trees.  High- 
tempo  operations  thus  lead  to  a  desynchronization  of  assessment  loops  between  the  joint 
and  subordinate  levels,  tending  to  devolve  to  the  tactical  or  immediate  execution  level. 


Ibid.,  53. 
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CHAPTER  5:  RECOMMENDATIONS 


Education  is  a  progressive  discovery  of  our  own  ignorance. 

-  Will  Durant 

The  DOD  uses  the  areas  of  DOTMLPF  (Doctrine,  Organization,  Training, 
Materiel,  Leadership,  Personnel,  Facilities)  to  identify  requirements  to  fill  capability 
gaps.  This  paper  will  use  these  same  areas,  when  applicable,  to  organize  the 
recommendations  for  implementing  changes  required  to  improve  assessment 
methodology. 

Doctrine 

Most  experts  agree  that  in  its  current  state,  military  assessment  is  ineffective  and 
in  need  of  significant  reform.  There  are  numerous  doctrinal  improvements  necessary  to 
properly  explain  and  detail  the  methodological  changes  required  to  improve  assessment 
practices.  The  most  crucial  change  to  make  is  adoption  of  the  civilian  practices  of 
separating  monitoring  and  evaluation  in  assessment  practices.  In  order  to  adapt  theory- 
driven  evaluation  correctly  to  military  operations  planning  and  execution,  assessment 
procedures  need  to  separate  the  monitoring  and  evaluation  functions  properly.  Doctrine 
must  clearly  define  that  the  purpose  of  monitoring  is  to  assess  the  execution  of  the  plan 
while  evaluation  is  to  critique  the  operational  design  selected  in  the  plan  to  determine  its 
continued  appropriateness  and  relevance.  At  the  same  time,  evaluation  will  subsume  the 
concepts  of  reframing  operational  design  elements  as  currently  related  in  Joint 
Publications.  This  allows  for  reframing  in  a  consistent  and  defined  process  as  part  of 
assessment  methodology.  Further,  the  military  can  use  formalized  evaluation  to  review 
and  reframe  operational  design  as  part  of  the  In-Progress  Review  (IPR)  of  the  Adaptive 
Planning  and  Execution  System.  Finally,  the  joint  staff  should  create  a  new  UJTL  task  to 
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mirror  OPS. 3. 1.3.  Conduct  Campaign  Assessment  that  defines  the  requirements  and 
metrics  necessary  to  conduct  evaluation.  Institutionalizing  the  separation  of  monitoring 
and  evaluation  will  help  solve  the  ongoing  issues  the  military  has  with  evaluating  the 
context  and  design  of  the  plan. 

The  next  step  of  this  reform  should  be  to  provide  clear,  detailed,  and 
comprehensive  doctrine,  terminology,  and  procedures  for  conducting  assessment.  To  do 
this,  the  joint  community  needs  to  collect  the  best  civilian  and  military  assessment 
practices,  methodologies,  techniques,  and  procedures,  and  publish  them  as  doctrine  in  a 
single  joint  publication  to  set  the  baseline  for  joint  assessment  operations.  This 
information  will  likely  be  too  large  for  inclusion  as  a  part  of  JP  3-0  and  JP  5-0  where  it 
currently  resides.  Instead,  the  Joint  Staff  should  separate  assessment  from  these 
publications  and  create  a  stand-alone  publication  to  hold  all  of  this  information. 
Currently,  JP  5-0  is  the  only  publication  with  a  single  volume  in  its  series  while  every 
other  numbered  series  has  at  least  one  subordinate  publication.  The  Joint  Staff  can 
recognize  the  importance  and  uniqueness  of  assessment  by  creating  a  JP  5-01  named 
Assessment. 

Another  important  requirement  is  to  develop  and  establish  a  standard  vocabulary 
and  process  for  assessment.  Clark  and  Cook  identified  this  same  fact  in  2008  and  went 
on  to  state,  “By  developing  a  standard  set  of  tools,  we  can  reduce  the  workload  of 
[assessment]  team’s  chiefs  by  eliminating  the  need  to  develop  and  maintain  their  own 
tools. Without  a  standard  vocabulary  and  methodology,  JFCs  cannot  share  practices 
and  experiences  effectively.  The  joint  community  must  cull  all  doctrine  documents  and 

*  Clinton  R.  Clark,  and  Timothy  J.  Cook,  "A  Practical  Approach  to  Effects-Based  Operational 
Assessment,"  Air  &  Space  Power  Journal  22,  no.  2  (Summer  2008),  98. 
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develop  a  lexicon  of  terms  and  definitions  that  eliminate  the  current  state  of  redundancy, 
disagreement,  and  common  but  confusing  synonyms.  By  establishing  a  common  and 
unified  vocabulary,  the  joint  community  will  have  the  groundwork  needed  to  move 
forward  with  the  improvement  of  assessment  practices. 

Another  key  issue  requiring  resolution  is  the  sharing  of  assessment  information 
between  military  organizational  levels.  The  joint  community  needs  to  address  this  issue 
by  giving  definitive  guidance  on  how  to  link  assessment  products  and  share  information 
between  organizations.  The  first  requirement  to  accomplish  this  is  to  set  up  a  standard 
assessment  plan  format  and  location.  Joint  doctrine  currently  does  neither  of  these 
things.  The  U.S.  assessment  handbook  states  that  an  assessment  plan  to  needs  be  detailed 
in  the  Operations  Plan  but  only  gives  a  very  rough  example  and  then  states  that  it  can  be 
included  as  an  appendix  to  the  operations  annex  or  under  the  reports  annex.  The  NATO 
assessment  handbook  goes  further  by  establishing  a  stand-alone  annex  in  the  Operations 
Plan  for  assessment."^  Unfortunately,  the  NATO  handbook  also  stops  short  of  providing  a 
standard  format  for  the  annex  or  for  listing  required  items. 

The  U.S.  military  needs  to  resolve  this  issue  by  developing  a  stand-alone  annex  in 
the  Operations  Plan  that  is  dedicated  to  the  assessment  plan.  Then,  it  needs  to  develop  a 
format  for  the  annex  with  a  list  of  requirements  for  monitoring  and  evaluating  the 
campaign  and  context  of  the  campaign.  Key  elements  that  must  be  included  are  how  the 
subordinate  commands  should  develop  their  assessment  plans  and  integrate  it  with  the 

^  Dave  LaRivee,  “Best  Practice  Guide  for  Conducting  Assessments  in  Counterinsurgencies,” 
Smallwarsjournal.com,  August  17,  2011,  http://smallwarsiournal.com/sites/default/files/838-larivee.pdf 
(accessed  August  12,  2013),  11. 

^  U.S.  Joint  Staff,  J-7,  Commander’s  Handbook  for  Assessment  Planning  and  Execution,  (Suffolk, 
VA:  Joint  Staff,  J-7,  Joint  and  Coalition  Warfighting,  September  9,  201 1),  IV -6. 

NATO  Operations  Assessment  Sub-Working  Group,  NATO  Operations  Assessment  Handbook, 
(Norfolk,  VA:  NATO  Supreme  Allied  Commander  Transformation,  2012),  2-7. 
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Joint  Force  headquarters.  It  should  also  list  the  organizational  level  that  will  be 
responsible  for  monitoring  each  item  and  creating  the  evaluation  and  recommendation. 
Finally,  doctrine  should  require  this  annex  to  specify  the  required  assessment  format, 
reports,  timeline,  and  other  key  assessment  procedures.  Standardizing  the  assessment 
plan  annex  better  enables  staffs  at  all  levels  to  link  their  assessment  plans  and  properly 
share  their  information. 

The  next  step  is  to  develop  a  process  for  integrating  the  assessment  plans  at  each 
level.  Currently,  each  level  creates  their  own  assessment  plan  and  tries  to  integrate  them 
with  other  levels,  but  there  is  no  formal  mechanism  for  conducting  this  process.  Much 
like  a  Joint  Force  Commander  receives  back-briefs  from  his  subordinates,  the  Joint 
assessment  team  should  interact  with  subordinate  units  as  they  develop  their  plans  and 
get  a  back-brief  on  its  linkage  to  the  higher  level.  Then,  the  higher  level  must  avoid  the 
assessment  fallacies  of  averaging  the  results  received  from  each  subordinate  or  allowing 
subordinates  to  mirror  the  metrics  created  by  the  higher  command.^  These  simple 
mathematical  tools  can  result  in  the  smoothing  out  of  unique  or  important  data  or  its  sub¬ 
summation  in  the  greater  mound  of  collected  data.  The  lower  level  commanders  have  a 
unique  view  of  ground  truth  that  senior  staffs  should  not  ignore  and  must  properly 
include  and  convey  in  the  final  product.  By  properly  establishing  and  integrating  the 
plans  between  the  multiple  levels  of  the  campaign,  it  will  be  possible  to  provide  the  best 
assessment  to  each  command  level  and  enable  them  to  make  timely  and  accurate 
decisions. 

The  final  concern  that  doctrine  must  address  in  assessment  sharing  is  how 

^  UNESCO  Bureau  of  Strategic  Planning,  Results-Based  Programming,  Management  and 
Monitoring  (RBM)  Approach  as  Applied  at  UNESCO  (Paris:  UNESCO,  2011),  18. 
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supporting  commanders  integrate  with  the  overall  assessment  plan.  The  Air  and 
Maritime  components  struggled  with  this  during  counterinsurgency  operations  in  both 
Iraq  and  Afghanistan  because  they  are  both  in  almost  complete  supporting  roles  with 
numerous  tasks  to  accomplish,  but  they  are  independently  generating  almost  no  effects 
toward  reaching  campaign  objectives.^  That  has  led  them  to  concentrate  on  MoP 
measurement  in  terms  of  number  of  aircraft  flown  or  tons  of  cargo  transported.  These  are 
important  measures,  but  provide  them  no  feedback  on  “If  they  are  doing  the  right 
things?”  The  Joint  Force  assessment  staff  needs  to  integrate  the  supporting  commanders 
with  the  supported  component’s  assessment  of  monitoring  and  evaluation  areas  so  they 

~j 

can  better  link  their  metrics  and  determine  if  they  should  alter  or  continue  the  plan.  It  is 
only  by  integrating  these  two  areas  that  supporting  commanders  will  be  able  to  properly 
assess  their  contribution  to  the  campaign. 

Assessment  doctrine  also  needs  updating  to  include  the  use  of  Structured 
Narrative  products.  The  current  described  practices  of  only  using  stoplight  charts  and 
quantitative  data  in  assessment  outputs  leads  to  misunderstandings  and  misinterpretations 
by  the  assessment  audience.  Including  some  textual  information  that  provides  context 
and  relevant  detail  to  the  quantitative  information  will  significantly  improve  assessment 
outputs.  The  use  of  Structured  Narrative  over  free  text  provides  a  needed  limitation  and 
rigor  to  the  textual  information  created  as  well  as  making  it  possible  to  electronically 
review  and  combine  the  data  to  create  the  best  possible  product  at  higher  echelons. 
Adding  Structured  Narrative  to  the  baseline  assessment  product  will  greatly  benefit  the 

®  Kirsten  Messer  and  Shane  Dougherty,  "A  New  Operational  Assessment  Paradigm;  Splitting  the 
Stoplights,"  Air  &  Space  Power  Journal  20,  no.  3  (Fall  2006),  66. 

’  Ibid.,  68. 
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successful  conveyance  of  assessment  information. 

Joint  doctrine  currently  lacks  an  established  mechanism  to  track  and  evaluate 
assumptions.  While  creating  a  new  mechanism  is  possible,  military  staffs  tend  to  respond 
better  to  mechanisms  and  processes  with  which  they  are  already  familiar.  Doctrine  does 
provide  tools  that  are  appropriate  for  identifying  and  monitoring  assumptions:  decision 
points  and  the  Commander’s  Critical  Information  Requirements  (CCIR).  A  CCIR  is  “an 
information  requirement  identified  by  the  commander  as  being  critical  to  facilitating 

o 

timely  decision  making.”  This  definition  is  fairly  broad,  but  staffs  tend  to  link  CCIRs  to 
decision  points  and  the  J-3  staff  uses  them  to  identify  when  plan  progressions  or  phase 
changes  can  occur.  In  addition  to  this  use,  the  planning  and  assessment  staffs  need  to 
return  CCIRs  to  their  broader  use  and  attach  one  to  every  key  assumption  that  is  made  in 
the  plan  to  ensure  it  is  properly  monitored  and  acted  upon  should  it  be  proven  false.  By 
using  the  decision  points  and  CCIR  construct  to  capture  the  assumptions,  assessment 
teams  can  ensure  that  the  commander  and  staff  do  not  drop  these  important  elements  as 
the  chaos  of  the  operational  environment  ensues  during  execution.  This  should  allow 
proper  identification  of  any  incorrect  assumptions  in  a  timely  manner  and  give  the 
commander  and  staff  the  cues  they  need  to  adapt  and  overcome  problems  before  they 
overwhelm  the  friendly  forces. 

Organization 

Once  the  revision  of  doctrine  to  lay  out  the  theoretical  basis  for  assessment 
improvement  occurs,  the  next  required  change  needed  is  in  the  assessment  organizations. 
According  to  the  J-7  Best  Practice  Focus  Paper  on  assessment,  there  is  no  standardized 

*  U.S.  Joint  Chiefs  of  Staff,  Joint  Operations,  Joint  Publication  3-0  (Washington  DC:  Joint  Chiefs 
of  Staff,  August  11,  2011),  GL-7. 
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directorate  or  composition  for  assessment  teams  (AT).^  Some  headquarters  locate  the  AT 
in  the  J-5  directorate  while  others  place  it  under  the  control  of  the  J-3.  Historically,  even 
the  J-2  and  Chief  of  Staff  have  been  used  to  spearhead  the  AT.  The  problem  with  the 
disparate  placement  of  the  AT  is  that  the  team  tends  to  take  on  the  characteristics  and 
emphasis  of  the  directorate  under  which  it  is  located. So,  if  it  is  under  the  J-5,  it  will 
tend  to  focus  more  on  the  long-term  effects  and  achievement  of  the  end-state,  while  if  it 
under  the  J-3  it  will  tend  to  focus  more  on  current  plan  execution  and  the  short-term 
adjustments  needed  to  ensure  that  the  plan  remains  successful. 

The  interesting  dichotomy  of  these  trends  is  that  they  exactly  match  the  difference 
in  concept  that  theory-driven  evaluation  holds  for  monitoring  and  evaluation.  So,  the  key 
to  solving  both  problems  is  deliberately  to  split  the  AT  into  two  separate,  but 
interdependent  groups  inside  of  the  Joint  Headquarters.  The  monitoring  group  will  be 
responsible  for  collecting  data,  analyzing  it  and  making  recommendations  for  changing 
the  current  plan  that  are  focused  on  the  executing  or  modifying  the  plan  to  achieve  the 
desired  effects,  objectives  and  end-states  of  the  current  approved  plan.  This  best  fits 
under  the  leadership  of  the  J-3  directorate,  but  will  be  composed  of  personnel  from  other 
directorates  of  the  headquarters  as  well.  It  will  develop  its  own  assessment  plan  using 
MOPs  and  MOEs  that  answer  the  two  big  questions  currently  stated  in  assessment 
doctrine:  “Is  the  JTF  doing  things  right?”  and  “Is  the  JTF  doing  the  right  things?”  This 
team  will  be  easy  to  create  and  execute  because  it  largely  falls  within  the  current 
doctrinal  guidance  and  methodology  employed  by  the  military  in  most  current  operations. 

^  U.S.  Joint  Staff,  J-7,  Deployable  Training  Division,  Insights  and  Best  Practices  Focus  Paper: 
Assessment,  (Washington  DC;  Joint  Staff,  J-7,  Deployable  Training  Division,  July  2013),  9. 

***  U.S.  Joint  Staff,  J-7,  Commander’s  Handbook  for  Assessment  Planning  and  Execution,  IV-8. 
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The  revolution  will  come  with  the  creation  of  a  separate  AT  responsible  for  taking 
a  more  holistic  view  of  the  context  and  design  of  the  campaign  or  operation.  Its  job  will 
be  to  evaluate  and  challenge  the  selected  elements  of  operational  design,  the  operational 
approach  and  assumptions  to  ensure  they  remain  accurate.  It  will  be  responsible  for 
developing  an  entirely  independent  assessment  matrix  to  assess  if  the  strategic  and 
military  direction  or  environments  are  remaining  the  same  or,  if  they  have  shifted,  how 
they  have  changed.  The  J-5  AT  will  also  revalidate  elements  of  the  operational  design 
selected  before  the  operation  began  as  a  means  to  identify  if  the  problem  requires 
reframing.  This  independent  and  wider  view  will  help  prevent  the  commander  from 
getting  lost  in  the  “tactical  weeds”  of  the  current  operation  and  allow  the  commander  to 
focus  on  the  real  goals  of  the  campaign  which  are  not  just  accomplishing  the  plan,  but 
rather,  reaching  the  current  and  most  advantageous  military  and  political  strategic  end- 
states. 

The  establishment  of  a  J-5  evaluation  assessment  team  focused  on  plan  evaluation 
will  also  provide  a  group  responsible  for  tracking  planning  assumptions  and  the 
assessment  metrics  themselves.  The  J-5  AT  will  fulfill  Sotire’s  recommendation  for  the 
development  of  a  metric  board  that  is  responsible  for  reviewing  all  metrics  on  a  regular 
basis  and  updating  them  as  necessary.  At  the  very  least,  an  assessment  team  should 
avoid  over-reliance  the  on  standard  metrics  that  are  available  in  guides  such  as  the 
Universal  Joint  Task  List.  While  these  guides  are  useful  for  establishing  a  baseline  for 
the  creation  and  tracking  of  metrics,  metrics  must  be  tailored  to  each  unique  campaign 

"  Robert  B.  Sotire,  Measuring  Performance  and  Effectiveness  in  Irregular  Warfare:  Preventing 
Dysfunctional  Behavior,  (Newport,  RI:  Naval  War  College,  2009), 
http://handle.dtic.mi1/100.2/ADA504933  (accessed  August  12,  2013),  17. 

LaRivee,  6. 
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plan  to  be  truly  useful  and  then  continuously  updated  and  refined  as  the  plan  is  executed. 
Most  importantly,  the  U.S.  assessment  handbook  that  describes  the  six  steps  of 
developing  an  assessment  plan  needs  updating  with  a  crucial  seventh  step  called 
“assessment  plan  review  and  update.”  The  purpose  of  this  step  is  to  define  the  process 
and  responsible  parties  for  reviewing  and  updating  the  assessment  process  as  the  plan 
proceeds.  Without  this  last  important  step,  especially  for  campaigns  that  continue  over 
weeks,  months,  or  years,  the  assessment  matrix  will  quickly  find  it  is  detached  from  the 
plan  and  no  longer  providing  meaningful  data  to  support  commander  decision-making. 

As  previously  discussed,  the  current  operations  which  serve  as  the  basis  for  most 
assessment  criticism  and  recommendations  are  protracted  warfare  campaigns.  These 
campaigns  are  of  a  very  different  nature  than  traditional  warfare  or  high  intensity 
conflicts.  Assessment  faces  a  different  set  of  challenges  during  traditional  warfare, 
largely  centered  on  providing  timely  and  relevant  assessments  that  can  keep  pace  with  the 
blistering  tempo  of  modem  U.S.  joint  operations  during  major  combat.  The  solution  to 
these  problems,  conveniently,  lies  in  the  previous  discussion  of  the  appropriate  separation 
of  the  assessment  processes  of  monitoring  and  evaluation.  The  definitions  and  conduct  of 
combat  analysis  closely  mirror  the  assessment  practices  of  tracking  if  the  operation  is  on 
plan  as  analyzed  by  the  civilian  definition  of  monitoring.  The  focus  is  on  tracking  the 
plan  and  making  sure  it  is  accomplished  correctly  and  driving  toward  the  stated  decisive 
points,  objectives,  and  end-states.  This  is  the  J-3  or  current  operations  execution  focus 
that  naturally  occurs  during  high-tempo  operations.  With  the  creation  of  a  separate  J-5 
evaluation  assessment  team,  it  will  be  able  to  focus  on  the  larger  and  longer-term  goals 

U.S.  Joint  Staff,  J-7,  Commander’s  Handbook  for  Assessment  Planning  and  Execution,  IV-6. 
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and  conduct  of  the  campaign  as  it  progresses,  and  will  not  be  mired  in  the  daily  or  hourly 
detail  of  the  plan  execution.  The  evaluation  cell  will  focus  on  the  larger  operating 
environment  issues  of  the  selected  elements  of  operational  design  that  are  liable  to 
change,  even  in  high-tempo  operations,  but  are  more  difficult  to  see  when  directly 
engaged  in  the  rapid  pace  of  current  operations. 

The  U.S.  has  a  poor  history  of  ending  traditional  warfare  operations  on  terms  truly 
favorable  to  the  U.S.  In  both  Gulf  War  conflicts,  the  U.S.  either  did  not  achieve  the 
termination  criteria  or  it  changed  during  the  operation.  At  the  end  of  the  1991  Gulf  War, 
the  U.S.  military  left  the  Iraqi  army  largely  intact  and  Saddam  Hussein  was  able  to 
maintain  his  stranglehold  on  the  Iraq  people  and  continue  to  threaten  his  neighbors  for 
another  decade.  By  the  end  of  the  2003,  Operation  IRAQI  FREEDOM  accomplished 
regime  change,  but  did  not  properly  assess  that  the  operating  environment  had  changed 
and  that  growing  insurgency  and  potential  civil  war  were  brewing.  Part  of  the  reason 
these  changes  went  unnoticed  in  both  operations  is  that  execution  of  the  current  operation 
consumed  the  attention  of  the  commanders.  By  separating  the  assessment  evaluation 
team  from  the  pull  of  the  J-3  during  high-tempo  operations,  the  command  will  have  a 
staff  dedicated  to  remaining  focused  on  the  operational  and  strategic  level  of  the 
operation. 

Training 

The  U.S.  and  NATO  assessment  handbooks  provide  a  good  initial  attempt  to 
capture  and  codify  assessment  techniques  and  procedures.  Unfortunately,  neither  of  them 
is  doctrinally  binding  nor  do  they  propose  best  techniques,  but  instead  list  available 
options  without  identifying  which  are  the  preferred  options.  Once  U.S.  doctrine  defines 
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and  endorses  the  best  assessment  methods,  the  joint  training  and  leader  development 


communities  should  develop  standards  and  training  processes  to  teach  it.  Numerous 
writers  and  publications  call  for  the  establishment  of  formal  assessment  training.  Even 
the  U.S.  assessment  handbook  takes  the  unusual  step  of  adding  an  entire  chapter  on 
training  recommendations  and  options.'"^  Interestingly,  it  does  not  go  to  the  length  of 
calling  for  the  development  of  a  formal  joint  training  course,  which  is  exactly  what  is 
required.  NATO  recognized  this  fact  and  established  an  assessment  training  course.  It 
established  the  pilot  “NATO  Operations  Assessment  Course”  which  was  held  in 
November  2012  at  the  National  Defense  Academy  in  Latvia  with  a  follow-on  course 
conducted  in  October  2013.'^  The  creation  of  this  course  fills  a  much  needed  assessment 
training  gap  and  should  be  mirrored  by  the  U.S.  military  and  attached  to  the  Joint  Forces 
Staff  College.  Once  established,  the  school  house  can  become  the  repository  for 
assessment  standards,  terminology,  and  processes.  Further,  it  will  give  the  U.S.  military 
a  place  for  experienced  assessors  to  pass  on  their  knowledge  and  expertise.  The  creation 
of  a  formal  assessment  course  would  be  incredibly  beneficial  to  the  practice  of 
assessment. 

Leader  Development 

The  purpose  of  assessment  is  to  provide  information  and  recommendations  to  the 
commander  to  use  in  his  or  her  decision-making  process.  This  means  that  the 
commander,  as  the  ultimate  user,  is  the  real  focus  of  assessment.  Unfortunately,  most 
commanders  have  poor  experiences  with  assessment  because  it  fails  to  live  up  to  their 


U.S.  Joint  Staff,  J-7,  Commander’s  Handbook  for  Assessment  Planning  and  Execution,  VI-2. 
NATO  Operations  Assessment  Sub-Working  Group,  6-58. 
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expectations^^  and  they  quickly  develop  a  distrust  of  it.  The  only  way  to  overcome  this 
deficiency  is  to  use  the  military  training  and  education  process  to  develop  commanders 
who  properly  understand  the  capabilities  and  limitations  of  assessment.  Military  officers 
need  to  understand  that  assessment  systems  are  never  perfect  and  often  riddled  with 
complexity  and  uncertainty  but,  if  assessments  properly  understand  and  account  for  these 
limitations,  the  staff  can  develop  a  good  recommendation.  Commanders  who 
understand  that  assessment  is  a  tool  and  not  a  panacea  are  more  likely  to  use  it  properly. 
Further,  if  they  understand  the  basic  vocabulary  and  processes  of  assessment,  they  will  be 
more  likely  to  use  it  correctly,  and  use  their  intuition  and  commander’s  judgment  to  fill  in 
the  gaps  with  which  all  assessment  plans  must  deal.  The  assessment  community  used 
this  solution  during  Operation  IRAQI  FREEDOM  in  2010,  when  it  created  a  two-day 
workshop  where  they  discussed  their  assessment  tools,  processes,  and  limitations  with  all 
key  leadership  in  the  coalition.  The  response  was  very  positive  and  commanders  felt 
much  better  prepared  to  use  the  assessment  information  they  were  provided.  The  U.S. 
military  should  add  a  robust  assessment  focus  to  JPME  II  training  to  provide  this 
knowledge  to  all  future  commanders.  By  properly  training  future  joint  commanders,  they 
will  be  best  positioned  to  use  and  exploit  the  assessment  information  they  are  provided. 

Personnel 

The  final  requirement  is  to  lay  down  baseline  placement  and  manning 
requirements  for  the  assessment  team  (AT).  Specifically  trained  personnel  must  fill  AT 
positions  in  order  to  ensure  the  best  possible  assessment  products.  This  means  that  ATs 

Jonathan  Schroden,  "Why  Operations  Assessment  Fails:  It's  Not  just  the  Metrics,"  Naval  War 
College  Review  64,  no.  4  (Autumn  201 1),  95. 

Sotire,  12. 

LaRivee,  8. 
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must  have  positions  for  assessment  experts  who  will  provide  the  necessary  mathematical 
and  research  expertise  to  prevent  the  team  from  making  logical  fallacies  in  their  products. 
Next,  the  AT  must  include  experts  from  all  directorates  to  provide  the  diverse  and  in- 
depth  analysis  required  for  an  accurate  assessment.  In  this  era  of  declining  budgets  and 
staff  manning,  it  may  not  be  possible  to  create  needed  manpower  billets  for  the  AT. 
Instead,  the  command  will  create  ATs  by  assigning  the  additional  duty  of  assessment  to 
personnel  from  each  directorate.  Finally,  as  previously  discussed,  the  military  must 
evenly  split  the  AT  into  two  parts,  and  align  them  with  the  J-3  and  J-5  directorates.  It  is 
only  by  defining  AT  manning,  filling  those  positions  with  assessment  experts,  and  then 
properly  aligning  the  teams  with  an  appropriate  Joint  Force  headquarters  directorate  that 
assessment  will  be  able  to  produce  correct  and  meaningful  advice  for  the  commander. 
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CHAPTER  6:  CONCLUSION 


Usually,  everyone  gets  it  wrong  at  the  beginning  of  a  war. 

-  Thomas  E.  Ricks,  The  Generals 

The  U.S.  military  is  likely  facing  a  significant  reduction  in  combat  capacity  and 
capability  as  it  enacts  the  measures  of  the  Budget  Control  Act  over  the  next  decade. 
Unfortunately,  the  threat  that  the  U.S.  military  faces  is  only  growing  in  this  increasingly 
complex  and  dangerous  world.  When  the  President  calls  upon  the  U.S.  military  to 
conduct  operations,  there  will  be  shrinking  room  for  error,  wrong  strategies,  or  misused 
forces.  This  increases  the  importance  of  all  four  functions  of  the  Joint  Force 
Commander:  plan,  prepare,  execute,  and  assess.  In  the  current  state  of  U.S.  joint 
doctrine,  assessment  is  ill-defined,  poorly  developed,  and  incorrectly  applied. 

This  paper  shows  that  current  assessment  doctrine  in  U.S.  Joint  Publications 
needs  significant  reform  by  drawing  on  civilian  theory  practices  to  standardize 
terminology  and  procedures  to  improve  its  successful  use  in  future  operations.  A  review 
of  the  civilian  theory  and  practice  of  assessment  demonstrated  that  military  assessment 
incorrectly  defines  the  process  of  monitoring  and  execution  in  such  a  way  that  they  both 
focus  on  measuring  progress  against  the  plan  rather  than  also  evaluating  the  adequacy  of 
the  plan  against  the  ever-changing  context.  Context  that  shifts  as  quickly  as  the  military 
operations  characterizes  war  and  conflicts  and  the  assessment  process  needs  to  use 
evaluation  of  the  context  to  ensure  that  the  military  plan  marries  up  with  current  political 
expectations.  As  part  of  this  process,  evaluation  will  appraise  and  constantly  improve  the 
assessment  plan  and  provide  monitoring  of  the  assumptions  used  in  the  plan.  Using  the 
civilian  practice  of  assessment  as  a  point  of  comparison  highlights  the  core  deficiencies 
in  the  current  U.S.  military  assessment  methodology. 
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A  review  of  the  current  state  of  assessment  doctrine  and  methodology  emphasized 
the  paucity  of  information,  and  lack  of  standards  and  procedures.  The  development  of 
assessment  in  the  U.S.  military  has  been  haphazard  and  did  not  benefit  from  the  extensive 
research  and  success  of  civilian  applications.  A  survey  of  U.S.  military  publications 
highlighted  that  current  doctrine  is  inadequate,  there  are  no  common  standards,  little  to 
no  training,  appropriate  subject  matter  experts  don’t  exist,  and  there  is  no  identification  of 
a  lead  directorate  for  assessment  functions.  These  factors  have  combined  with  the 
challenges  of  sharing  assessments  between  organizational  levels,  the  lack  of  product 
uniformity,  and  the  unique  requirements  of  traditional  warfare  assessment  to  explain  why 
the  current  practice  of  assessment  is  not  meeting  the  JFC’s  requirements. 

From  the  two  major  identified  root  deficiencies  of  the  current  U.S.  military 
assessment  methodology  and  doctrine,  the  thesis  provides  several  recommendations  using 
the  DOTMLPF  format.  First,  doctrine  needs  to  inculcate  the  civilian  theory  and  practices 
of  separating  monitoring  and  evaluation  along  with  providing  detailed  guidance  on 
consistent  assessment  methodology.  Commands  can  improve  the  organization  of 
assessment  teams  by  separating  them  between  the  J-3  and  J-5  cells.  Training 
solidification  requires  the  establishment  of  a  U.S.  assessment  training  course.  Leaders 
are  the  ultimate  recipient  of  assessment  products  and  need  training  in  assessment 
practices  as  part  of  JPME II.  Finally,  the  Joint  Staff  needs  to  establish  a  baseline  for 
personnel  skills  and  background  that  comprise  an  effective  assessment  team.  By 
adopting  these  new  assessment  practices,  the  U.S.  military  can  dramatically  improve  its 
assessment  practices. 

Doctrine  charges  the  Joint  Force  Commander  with  planning,  preparing,  executing. 
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and  assessing  military  campaigns  or  operations.  Just  like  the  example  of  General 
McChrystal  in  his  evaluation  of  the  poor  assessment  practices  he  found  upon  his  arrival 
in  Afghanistan  after  eight  years  of  war,  commanders,  staffs,  and  national  leaders  have 
been  unhappy  with  the  assessment  products  created  to  evaluate  the  campaigns  over  the 
past  few  decades.  The  failure  of  assessment  methodology  is  largely  due  to  the  lack  of 
adequate  common  guidance  and  negligent  misapplication  of  successful  equivalent 
civilian  theory  and  practices  into  the  military.  By  making  several  improvements  in 
assessment  methodology  and  giving  it  the  attention  it  deserves  as  one  of  the  primary 
responsibilities  of  military  leaders,  the  U.S.  military  will  be  able  to  create  an  assessment 
system  that  provides  current,  relevant,  timely,  correct,  and  appropriate  assessment 
information  which  will  allow  the  commander  to  improve  upon  military  operations. 
Without  making  these  changes,  the  U.S.  military  will  continue  failing  to  understand  and 
adapt  to  the  ever-changing  strategic  context,  thus  wasting  national  resources  or  even 
failing  to  accomplish  its  strategic  objectives. 
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APPENDIX  1:  STRUCTURED  NARRATIVE  EXAMPUE 


The  medical  community  is  leading  the  development  and  use  of  Structured 
Narrative.  The  following  examples  draw  from  that  field  and  show  methods  of  input  and 
display.  Figure  1  shows  the  simplest  example  of  Structured  Narrative  input  where  the 
user  can  only  select  between  preset  options  in  each  field.  This  form  does  not  allow  the 
input  of  free  text. 


Figure  1.  Example  of  Structured  Narrative  entry  not  allowing  text. 

Figure  2  shows  a  more  complex  example  of  Structured  Narrative  input  where  the  user  can 
enter  a  combination  of  preset  options  and  free  text  in  different  fields.  By  using  the 
separated  fields  and  defining  the  category  of  free  text  entered,  it  is  possible  to  use 
computers  to  aggregate,  analyze,  and  even  search  all  data  entered. 


^  BioMed  Central,  "BMC  Medical  Informatics  &  Decision  Making,"  BioMed  Central, 
http://www.biomedcentral.eom/1472-6947/6/29  (accessed  February  27,  2014). 
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Figure  2.  Example  of  Structured  Narratives  with  text  and  preset  options. 


Finally,  once  the  Structured  Narrative  has  been  entered,  it  can  be  displayed  in  four 


different  formats:  graphical,  definition,  structured  text  and  tabular. 


Figure  3.  Example  of  the  four  display  formats  of  Structured  Narrative. 


^  Ibid. 

^  Semantic  Community,  “Be  Informed  4  -  They  Really  Got  It,”  Semantic  Community, 
http://semanticommunitv.info/AOL  Government/Be  Informed  4  -  They  really  got  it  (accessed 
February  27,  2014). 
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